Available formats: content-negotiated html turtle (see SIOC for the vocabulary)
Back to channel and daily index: content-negotiated html turtle
These logs are provided as an experiment in indexing discussions using IRCHub.py, Irc2RDF.hs, and SIOC.
| 02:24:29 | <mightybyte> | I have a Happstack app that's using 273 megs of RAM, but the state files are only 1 meg. Anyone have an idea what might be going on here? |
| 02:25:19 | <gwern> | mightybyte: well, profiling would help I'd imagine |
| 02:26:44 | <mightybyte> | With the ghc profiler? |
| 02:27:43 | <gwern> | sure why not |
| 02:28:22 | <mightybyte> | Hmmm, on a live site? |
| 02:28:44 | <gwern> | it's not like a profiled binary will eat visitors' kittens |
| 02:28:45 | <mightybyte> | I'm also getting a fair amount of these errors: |
| 02:28:49 | <mightybyte> | HTTP request failed with: <socket: 14>: hLookAhead: resource vanished (Connection reset by peer) |
| 02:28:54 | <mightybyte> | HTTP request failed with: <socket: 13>: hPutBuf: resource vanished (Broken pipe) |
| 02:29:10 | <mightybyte> | noscrolls: <socket: 10>: hClose: resource vanished (Broken pipe) |
| 02:29:34 | <mightybyte> | That last one is sent to stderr. |
| 02:30:07 | <gwern> | yeah, I never did figure out what those were |
| 02:30:33 | <gwern> | couldn't reproduce them, which makes me think they're from a wonky client - maybe a dropped connection or something |
| 02:41:28 | <jimmyjaz114> | I want to build my blog using happstack where is a good place to start |
| 02:41:31 | <jimmyjaz114> | ? |
| 02:42:34 | <gwern> | look at everything with the word 'blog' on hackage? |
| 02:47:06 | <mightybyte> | gwern: Yeah, I get those errors regularly, but so far I haven't figured out how to reproduce them. |
| 03:01:09 | <sm> | same here |
| 03:02:23 | <sm> | jimmyjaz114: perhaps http://gregorycollins.net/posts/2009/03/28/building-a-website-part-1 |
| 04:04:07 | <mightybyte> | gwern: Hmmm, do I have to compile all the happstack libraries with profiling too? |
| 04:24:07 | <sm> | mightybyte: pretty sure you do |
| 10:18:16 | <HugoDaniel> | hi |
| 10:19:32 | <HugoDaniel> | when is 0.3 getting released ? |
| 10:39:05 | <mightybyte> | I tried compiling my app with -prof to do some memory usage debugging. |
| 10:39:17 | <mightybyte> | I got: "Perhaps you haven't installed the profiling libraries for package `happstack-data-0.3.1'" |
| 10:40:05 | <mightybyte> | I'm new to ghc profiling in general. Does Happstack have a mechanism for building with profiling turned on? |
| 11:26:02 | <sm> | mightybyte: generally you do cabal install --reinstall -p |
| 11:26:49 | <sm> | this will probably miss some dependencies, or report some missing thing or other, in which case add that lib's name, and repeat |
| 11:34:50 | <sm> | ACTION crashes |
| 11:56:03 | <mightybyte> | sm: Does -prof incur any kind of significant overhead that would make it inappropriate for use in production situations (when not executing with +RTS)? |
| 13:00:25 | <gwern> | mightybyte: 1) yes; 2) the prof overhead is minimal, no more than a few % surely |
| 13:33:30 | <mightybyte> | Wow, it looks like compiling with profiling will require a reinstall of every dependency. |
| 14:53:21 | <gwern> | yeah. profiling sucks that way |
| 15:00:54 | <mightybyte> | This is interesting. It looks like my app has expanded to consume all the available memory, and then levelled off. |
| 15:18:02 | <gwern> | ... |
| 15:18:17 | <gwern> | ACTION wonders what memory you would expect it to consume after that |
| 16:11:29 | <mightybyte> | Well, all the data in _local consumes 1 meg, so I wouldn't expect it to level off at 300 megs. |
| 17:59:08 | <mightybyte> | In a profiling run of just starting my app (with an offline copy of real user-generated state data), 77% of CPU and 80% of alloc were used by Happstack.Data.IxSet.flatten. |
| 18:00:02 | <mightybyte> | I just started the app, and shut it down. No requests were made to the server. |
| 18:13:23 | <gwern> | so you've learned something, you're doing arrays wrong |
| 18:14:45 | <mightybyte> | What do you mean? flatten is only called from IxSet.change. |
| 18:14:59 | <TomMD> | Does it really matter the startup and shutdown cost breakdown? So long as startup is 'fast enough' then who cares? The real concern is efficiently serving requests once the server has started. |
| 18:15:23 | <gwern> | if I only had a few megs of data and array manipulation is the vast lion's share of my executable's time, I'd be worried |
| 18:15:23 | <mightybyte> | It's not about startup CPU cost, it's about memory cost. |
| 18:15:44 | <TomMD> | Yeah, sorry. I just saw your comment that its consuming 300 MB to read in a 1 MB store? owch. |
| 18:15:45 | <gwern> | at the very least, I'd hope I had a good explanation. |
| 18:16:32 | <mightybyte> | Well, the first explanation could be that I didn't have any checkpoints, so as I understand it, all the transactions were played from initialState. |
| 18:17:31 | <mightybyte> | In my mind, that would explain the activity, but I would that would not imply such a large memory footprint. |
| 18:17:44 | <TomMD> | Another possibility is that the code reading in the data builds up a huge number of thunks - thus strictness annotations would be needed. |
| 18:18:07 | <TomMD> | Have you looked at Happstack.Data.IxSet.flatten? |
| 18:18:23 | <mightybyte> | Briefly |
| 18:19:24 | <mightybyte> | Is the state automatically loaded from the most recent checkpoint? |
| 18:23:35 | <mightybyte> | Aha, I profiled a startup again after I created a checkpoint, and it only used 27 megs of RAM. |
| 18:24:37 | <mightybyte> | This time flatten was 60% time and 63% alloc. |
| 18:26:29 | <mightybyte> | So it appears that playing through transactions is costly in space. |
| 18:26:39 | <mightybyte> | ...at least transactions involving IxSet. |
| 18:44:06 | <mightybyte> | That doesn't bode well for long-term happstack uptimes. |
| 18:50:35 | <TomMD> | Can't they checkpoint at any time? |
| 18:51:55 | <mightybyte> | Yeah, but a running site is characterized by transactions. |
| 18:53:05 | <mightybyte> | So if update transactions are "leaking" memory, then you're eventually going to be forced to restart the app even if your total state size easily fits in memory. |
| 18:54:59 | <TomMD> | mightybyte: I thought what was happening is the replaying of stored transactions builds lots of thunks before evaluating. This shouldn't (wouldn't?) happen on a running site because those thunks need to be evaluated regularly to display the content in question. |
| 18:55:54 | <mightybyte> | I'm not sure precisely what's happening yet. |
| 18:56:22 | <mightybyte> | But flatten is using the most memory and it is only called by IxSet.change. |
| 18:57:58 | <mightybyte> | So state updates are what seem to be consuming the most memory. |
| 18:58:21 | <mightybyte> | And unless you're just serving static content, those updates over time will eat up your memory. |
| 18:59:14 | <mightybyte> | And it becomes very important to checkpoint regularly to eliminate the accumulated leakage. |
| 19:07:31 | <mightybyte> | But with those checkpoints comes the need to restart. |
| 19:19:31 | <TomMD> | Why do you need to restart after checkpointing? |
| 19:23:42 | <mightybyte> | Because the memory has leaked and the point of checkpointing (at least in my case) is to keep memory usage in check. |
| 19:25:59 | <mightybyte> | Hmmmm, compiling with -O makes a big difference too. |
| 19:27:22 | <mightybyte> | After checkpointing (but without -O), memory usage was 45 megs. With -O, it dropped to 13 megs. |
| 19:56:17 | <luntain> | these FromData instances are just boiler plate |
| 20:17:14 | <sm> | interesting (memory usage) |
| 21:05:11 | <gwern> | mightybyte: why do you ever run non -02 stuff? |
| 21:09:12 | <mightybyte> | gwern: Didn't really think about it (and in retrospect, know better). |
Back to channel and daily index: content-negotiated html turtle