Changelog for nzbperl -- the perl based nzb downloader Please visit http://noisybox.net/computers/nzbperl/ for more information -------------------------------------------------------------------------- 2005-02-26 * Truncate displayed filename if too wide for screen -------------------------------------------------------------------------- 2005-02-05 - * Added some fixes around handling many missing parts/files more gracefully * Got preliminary bandwidth and throttling working...still needs work. -------------------------------------------------------------------------- 2005-01-29 - * Rebuilt the keepbroken and keepbrokenbin functionality. * Rebuilt skipping over downloads when they exist on disk. * Fixed issues around dlcreate, dlrelative, and dlcreategrp. * Started rebuilding the main display part of the screen... -------------------------------------------------------------------------- 2005-01-28 - * Got file segment merging and decoding working! Pretty great to now see those several connections slurp down a single file... * Fixed a bug where the queue would skip over partial download files when a completed file was spliced from the nzb array. * Rebuilt skipping over already downloaded files -- now works again. * Implmented removing of segment files when a forced quit happens. * It was *not* easy, but I got graceful shutdown going after all segments from all files from all nzbs are finished. Yeesh. -------------------------------------------------------------------------- 2005-01-22 - * Still a very long ways away, but I succeeded in getting a single connection to download all parts in a file. Had to manually assemble, but things seem reasonable so far. -------------------------------------------------------------------------- 2005-01-21 - * Started working on 0.7.0. First steps -- encapsulate some of the common stuff (File, Segment, Connection) into packages. Hope to include some unit tests before too long. * Eeek...this was a lot of work, lots of changes, didn't even start test work. Still a long ways away from even downloading files with the new version, but it'll get there... -------------------------------------------------------------------------- 2005-12-26 - * Fixed bug that allowed --server=:119 to be considered a valid server and port parameter. This is no longer allowed. * Remove whitespace around first equals when pulling options from configuration file. Without this, it was possible for whitespace to be included in parameter values, and causing weird problems, like 'Inappropriate ioctl for device' when trying to connect to server. * Explicitly remove whitespace around $server, $socks_server, $http_proxy_server. * Implement trimWS for reuse. -------------------------------------------------------------------------- 2005-12-19 - * First implementation of disk free space check in place via --diskfree parameter. The checking is quite basic: before a connection will take on a new file, the disk is checked. This will really slow things down if/when processing many small files. There's also a rare case where the target dirs could be on different volumes, but nzbperl will *not* skip ahead in the queue (after all, it's a queue). When nzbperl detects that the disk for the next dequeue exceeds the free space, it will block all other files on all other (possible) volumes until the space is free. Connections are left open while waiting for free space, and once free space is available again, any connections that have had server disconnects will go through their regular reconnect delay cycle. It's probably not perfect, but I think it's quite good with minimal effort. * Changed the display stuff a little to support disk full conditions. * To test the above without lugging around gigs of data, I also added the 'diskfree' command to the remote control handler. The diskfree param can thus be changed remotely, if you like that sort of thing. -------------------------------------------------------------------------- 2005-12-18 - * Consolidated some duplicated code related to figuring out where the downloaded file on disk will go. This will help facilitate free space checking (reuse good, duplication bad). -------------------------------------------------------------------------- 2005-12-15 - * Gracefully handle up-to-date check failures. Previously, error messages would be displayed if the remote fetch didn't work. -------------------------------------------------------------------------- 2005-12-11 - * There was a report on the list about quotes breaking the --postnzb and --postdec shell commands. The patch submitted didn't seem to work as I expected, so I tweaked the command and do explicit escaping now. * Incorporated Miquel's patch to sort the segments during nzb parsing. This prevents false "bad" detection in the sanity checking routines. Without this patch, nzbperl would report broken nzbs when the segments weren't in numeric order. * Added a check at options parse time that confirms that the download directory is writable. If not, the program will bomb with a nice error message now. * Confirm that nzb path is writable when the temp filename is built when using --dlrelative. If not, log an error message and let the thing fail. -------------------------------------------------------------------------- 2005-11-17 - * So it seems that I screwed up the environment variable thing when adding that semicolon to make the rename item in the faq work properly. I spent about an hour tonight figuring out more details. It seems that a nice combination of 'export' *and* the semicolons will provide the best of both worlds. I was a bit concerned about the multithread/proc thing causing conflicts, but environments don't propagate to parents, so I finally think we're ok. -------------------------------------------------------------------------- 2005-11-14 - Releasing 0.6.7 today * Implemented --dthreadct option to allow the number of decoder threads to be specified. I still find it unusual that 1 isn't enough for some people, but oh well...at least it's configurable now. * After building --dthreadct it didn't seem to make much sense anymore to have the --nothread option (since nothread is really --dthreadct 0), so I removed the --nothread option and mapped the functionality in a function that examines the decoder thread count. * Fixed a serious bug (in my opinion, although users haven't complained) that caused the status messages to be redrawn unnecessarily. This caused a LOT of thrashing with multiple connections when all but one connection was idle (ie. at the end of a download) -------------------------------------------------------------------------- 2005-11-12 - * Fixed bug in remote client command parsing/handling that caused warnings to muck up the display. I was somehow trying remove from the buffer kinda twice. * Started a faq page on the website. * Added semicolons between the env vars on --postnzb running. * Added "speed" command to remote control protocol. The speed can now be explicitly set with a single remote command. -------------------------------------------------------------------------- 2005-11-10 - * Added --ifilter option to do an inverse regex filter on subject lines. Sounds maybe a little silly, but it can be useful when you only want to target a subset of files that can't be easily made into a --filter. Can also be used with --filter. -------------------------------------------------------------------------- 2005-11-02 - * Changed part of the nzb file parsing so that the segment number inside the nzb isn't used as an array index. That was the only place it was being used, and it really doesn't provide much value. I tested this out by trying all kinds of numeric, out of sequence, negative, bogus segment number values, and files still download. -------------------------------------------------------------------------- 2005-10-25 - * Thomas on the list provided a couple fixes to undef/uninit stuff, and I incorporated it into the dev version. -------------------------------------------------------------------------- 2005-10-16 - * Used Eric's patch to use $hchar so that nzbperl now uses the full terminal when drawing. Previously, it would only use a portion of the terminal...now it will use the whole thing. I suspect that most people will just have more log messages...which is probably a good thing. -------------------------------------------------------------------------- 2005-10-09 - * Wrapped the use threads and use Thread::Queue in an eval to prevent compilation errors. I didn't realize that this was happening (because I don't have a nonthreaded Perl). * If --decodelog isn't specified, stderr and stdout are now redirected to /dev/null to prevent broken files from spewing stderr into my nice display (thanks to Saul for pointing this out). Apparently uudeview with -q doesn't prevent broken/corrupt files from outputting on stderr, so they're now redirected. You *should* be using --decodelog anyway, though. :) -------------------------------------------------------------------------- 2005-10-08 - * Applied for a sourceforge account for nzbperl. Hope to use it mostly for CVS and email lists...and this can hopefully help users that want cutting edge changes as they're committed. -------------------------------------------------------------------------- 2005-09-17 - * Removed hard coded ENV{PATH} that was left over from the taint stuff. This caused uudeview to not be found when it was in a nonstandard location but still on the path. Screwy taint crap. * Fixed problem with dlpath default when --dlrelative specified (thanks to Thomas for patch) * Promoted $DECODE_DBG_FILE to a commandline option, which defaults to nothing. Without this, multiple could not run nzbperl at the same time because of permissions issues. * Check for writability of $DECODE_DBG_FILE during option parsing before starting (to prevent full download from happening) (Thanks to Thomas for related patch). -------------------------------------------------------------------------- 2005-09-11 - Releasing 0.6.6 today * I've been doing a really really poor job of keeping up with the changelog, even though I have been doing work on nzbperl from time to time...which kinda includes: * Removed -T (taint) from top usage. I understand the security implications, but I just don't feel like dicking with it right now. I'm welcoming patches related to untainting after 0.6.6 is released. * Promoted recv_chunksize to parameter (--chunksize) * Reworked the nzb parsing/sorting so that the last file in a nzb file is marked as such. This was required to facilitate the --postnzb option...which was ultimately a big pain in the ass to implement. :) * --queuedir now requires an absolute pathname. * --dlpath now requires an absolute pathname. * Added new options to the sample rc file. * Ok, so it's still a little rushed, but I'm going to release 0.6.6 today anyway... -------------------------------------------------------------------------- 2005-08-24 - * More work on -T and -w (taint and warn) complaints. As part of this, I have to set ENV to a known set of values, which could (will?) break some of the nice path searching for uudeview on non-standard setups. I think this will be ok, however, when uudeview can be specified in the configuration. * Looks like I screwed up the nothread option in the 0.6.5 release. I didn't properly wrap all of the thread/thread queue checks with checks around the $nothread variable. As a result, people trying to use the --nothread option are likely all running into issues. I *think* I've got them all properly wrapped now...but again, it's hard to test because I don't run a nonthreaded Perl...and I'm too lazy to install one. -------------------------------------------------------------------------- 2005-08-24 - * Ouch...I'm getting all kinds of warnings. I should have expected this, but it's mostly housekeeping now. Not spending much time on this tonite. -------------------------------------------------------------------------- 2005-08-21 - * Added -w and -T options to perl to get warnings and taints. Heh. "Taint". -------------------------------------------------------------------------- 2005-08-20 - * Spent several hours running with --forever and --queuedir running through nc to troubleshoot a few nasty problems that were cropping up. I think I've got most of them fixed, but the --forever thing got to be way more complicated than I expected (especially because of the nzb file queue, all that created all new kinds of entry points that were never previously expected). * Idle connections will show show [closed] after remote disco. * Improved reconnection robustness in various places, I think. :) * Put an additional level of hashref in the nzbfiles to track what the currently "active" file is to facilitate post nzb file completion program running. -------------------------------------------------------------------------- 2005-08-19 - Continuing 0.6.6 * Started builidng the --forever option to prevent nzbperl from ever dying. -------------------------------------------------------------------------- 2005-08-16 - Started working on 0.6.6 * Added --forever option that keeps nzbperl around forever. The command must be used with either --rcport or --queuedir (which are the only 2 options that get data into nzbperl) * Changed remote "queue" command to "enqueue" for clarity. * Started working on documentation for the remote control protocol stuff. http://noisybox.net/computers/nzbperl/nzbperl_remote.txt -------------------------------------------------------------------------- 2005-08-07 - Releasing 0.6.5 today * Reworked the commandline handling so that error messages can be uniformly displayed at the BOTTOM of the help screen dump when there are option problems. Because the help screen is long, people were having problems realizing that there were error messages in there. Should be MUCH clearer now...and I also capture GetOptions err msgs for consistent display. * Use Cwd for getting current directory, instead of using ./. Absolute paths are just easier to work with, and apparently Cwd is extremely command and portable. * Finished up the --postdec option functionality. It's now possible to have nzbperl call an external program after decoding each file. The primary intention is to allow people to build their own add-on binary assemblers, or whatever. * Added postdec and relevant notes to the sample nzbperlrc file. * Incorporated Thomas's patch to support IPv6 via --ipv6 commandline option. I have NO way to test this, so until I get feedback from people actually using ipv6, it's considered EXPERIMENTAL. All I can say is that it looks like I didn't break ipv4 anyway. :p * Added --ipv6 commandline option to help screen and sample rc file. * Added --rcport to usage screen (was never included) * Added --rcport to sample nzbperlrc file -------------------------------------------------------------------------- 2005-07-31 - * Fixed the 100% CPU usage situation when all connections are disconnected (like when waiting for all to reconnect). * Ensure that nzb parsing problems are shown when using queuedir and a single file is slurped in but no nzb given on commandline. Previously, it was hard to tell what happened...at least the errors are shown on screen now. * Started working on --postdec option which allows a program to be run after each file is decoded. Not yet complete. -------------------------------------------------------------------------- 2005-07-22 - * Added quick exit for drawScreenAndHandle keys if running in daemon mode (users said it was causing problems...tho it never did for me) -------------------------------------------------------------------------- 2005-07-10 - * Refactored a bunch of crap to facilitate --nothread option. It probably needs more testing, but it should be possible now to specify --nothread to disable the decoding thread and to decode parts "inline". Nice for people who don't have Perl compiled with threads. * Added --nothread option to help screen and to sample nzbperlrc * Apparently --daemon was missing from the nzbperlrc sample file. * Implemented "queue" remote control command. Allows any local file to be added to the nzb file queue. * Added completed items to summary command response. * Added display of number of remote clients to header screen part. -------------------------------------------------------------------------- 2005-07-04 - * Continued refactoring of gigantic subs. Still has room for improvement, but I suppose it's better than before. * Added missing/new options to the sample nzbperlrc * Built "ping" remote control handling * Built "summary" remote control handling -------------------------------------------------------------------------- 2005-07-06 - * Fixed a bug related to idle connections with nothing else to do being closed by the server and attempting to write the close statement from the server into an undefined file handle. Yes, that's a mouthful, sorry. For my own sanity: 503 Timeout after 900 seconds, closing connection. is what at least one server spits back when a connection is idle too long. This is all going to need reworking when the multi server thing comes around...until then, it's just a fix. * Started some refactoring of the gigantic doReceiverPart sub that was simply too fucking big. -------------------------------------------------------------------------- 2005-07-04 - * Changed close calls to undef calls for the temp filehandles...in trying to troubleshoot an issue that at least 1 user has...where the tmpfile filehandle ref goes bye bye (or turns into an empty string in some magic way). Still not sure why this crops up, but I haven't been able to recreate it. -------------------------------------------------------------------------- 2005-07-02 - Started working on new version (0.6.5) * Added improved error reporting when XML parsing fails. It should be pretty clear now when the nzb file is invalid. * Improved display handling during queued nzb file parsing. * Added seconds to the log file timestamps * Abort startup if there are unrecognized options * Added --dlcreategrp option that will create target directories that correspond to the group name that the file was downloaded from. * Started laying in the foundations for socket-based remote control. * Implemented the "keys" RC command. It interprets a series of characters as though they were pressed from the console. This provides certain nice things, like being able to remotely quit the program or dynamically/programmatically change download speeds...which are both things that users have asked for. :) -------------------------------------------------------------------------- 2005-06-20 - Version 0.6.4 released today! * No changes, just the release (maam) -------------------------------------------------------------------------- 2005-06-12 - * Fixed an evil bug related to connection logic. The parameter passing into the createSingleConnection was inconsistent, likely due to new use of IO::Socket::INET. Should be fixed/consistent now. -------------------------------------------------------------------------- 2005-06-11 - * Added --queuedir to the help screen (forgot to do that previously) * My news provider might have their shit together now, so doing some additional testing... -------------------------------------------------------------------------- 2005-06-03 - * Incorporated John's one liner patch to fix the progress bar length bug when the file sizes in the nzb file are wrong. Previously, it could allow the progress bar symbols to fill the screen and generally muck things up. There's an explicit check and cap for this now. Thanks John! -------------------------------------------------------------------------- 2005-05-30 - * Added a per-connection sequence id to the temp filenaming. For very small files, it was possible that the same filename would get used twice, which would cause a mess with the threaded decoding. Temp filenames should now be truly unique per-run. -------------------------------------------------------------------------- 2005-05-29 - * Added in the ability to queue nzb files from a directory (essentially a directory watch). The option is --queuedir . nzbperl will look for new, unprocessed *.nzb files in the queuedir and will load them at runtime. Because the default nzb validity checking prompts the user interactively, using --queuedir requires either --insane or --dropbad options. When a connection becomes idle and there are new, unprocessed nzb files in queue, one will be popped off the queue and its files and parts will be processed. A few related notes: When using the nzb file queue, the files remaining, bytes remaining, percentage complete, and ETA will all be wrong. When a new file is slurped up from the nzb directory queue, the values will be updated...but they certainly no longer reflect the overall run. ...they simply cannot be updated accurately until we read in the next nzb files...this is the price you pay for queueing up new nzb files. Sorry. * Updated the display routine to indicate how many nzb files are in queue when running with the --queuedir option. * Tested --queuedir with --dlrelative (works) -------------------------------------------------------------------------- 2005-05-25 - * Updated the website to talk about how to install modules via CPAN. * Added debian packaging to the TODO (mentioned a few times by users) * Changed pc() routine to default to plain white (to help facilitate unifying output techniques. -------------------------------------------------------------------------- 2005-05-22 - * Ok, socks auth is working...I just had dante misconfigured. --socks_user and --socks_passwd are working. * Added in interactive password entry for socks password (so that it's not required on the commandline or in the config file). This prevents divulging the pw to other users (in ps or on filesystem). * Changed --socks_user and --socks_passwd to --proxy_user and --proxy_passwd so that I can reuse these for http proxy...which I'm setting up soon. Ok, Net::HTTPTunnel is completely broken. * Reworked some of the socket handling stuff (send/recv) and it seems that I've gotten the http tunnel stuff working. Neet. * Ran a test using http tunnel through ssl...and woah, it seems to have worked. * Ran a test using socks through ssl...and again, it seems to have worked. * Added new commandline args and descriptions to the sample nzbperlrc file. * Incorporated Frederic's sweet lookin' ANSI graphics patch. The border and the progress bar are now drawn with ANSI lines, which can be disabled with the new --noansi option. I need to better understand how the ANSI escape codes are generated...so I can use them more. If anybody knows, please help out. -------------------------------------------------------------------------- 2005-05-21 - * Continued reviewing Neil's changes...hope to test new features pretty soon. * Added new commandline options for Neil's changes to the --help text. * Tested --ssl with stunnel. Seems to be working...although certs aren't being validated yet. Program segfaults on exit?. * Tested --socks_server with danted. Seems to be working initially. * Added in socks authentication (to the socks server), but danted doesn't seem to like it... -------------------------------------------------------------------------- 2005-05-19 - Version 0.6.4 started/continued * Incorporated Neil's patch which includes some excellent changes: * Support for SSL * Support for SOCKS proxy * Support for HTTP proxy * Fix to isbroken checking for broken files * Create download directory names based on nzb filename * Use getservbyname to resolve "standard" port numbers (instead of just hard coding them) * Additional code cleanup/improvements * Changed conditional 'use'es to be wrapped in eval block. -------------------------------------------------------------------------- 2005-04-17 - Version 0.6.3 released today! * Added ability to automatically check for newer versions on startup. It uses LWP::Simple (which I think most people have installed), but I wrapped it with some additional error handling to prevent the lack of LWP::Simple from causing the app to bomb out. If it's not installed, it will just skip the update check. If a newer version is available, it will display a short message at startup and delay for a few seconds. * Added --noupdate commandline option to disable to up-to-date checking. This option is disabled by default, so that the default behavior is to check for new versions. * Moved filtering, sorting, and skipping to outside the parseNZB routine. These operations needed to happen now after all nzb files have been read and parsed. * Fixed a display bug that I think was introduced when showDecodingStartStopMsg was removed yesterday. -------------------------------------------------------------------------- 2005-04-16 - * Started work on threaded decoder. Apparently users really don't like to have their downloads stalled during a decode. Impatient dorks. :) The users have spoken. (later) Ok, the threaded decode is mostly working. Needs a bit more testing, but seems to be doing what it should. * Removed the showDecodingStartStopMsg function because it's no longer being used. The decoder thread only outputs to the statMsg/log file. * Help screen now shows what server it's connected to. Of course, this may introduce other issues when multiple servers are supported, but until then, it's still helpful. * Capped the throttling sleep duration at 1s, to prevent the app from becoming totally unresponsive at slower speeds. Of course, this now means that there is a minimum speed at which most people can download, but I think that 1s sleeps between reads should be enough for almost everybody. * Started cleaning up some of the print statements to pc() calls in preparation of the silent mode (via --daemon option) * Also cleaned up the terminal Tgoto stuff to call a cursorPos function, in prep for the daemon mode. This is probably a good thing anyway, as it makes all the calls consistent and could lend itself to better terminal handling on other systems...? * Added --daemon commandline option. Daemon mode should work now. I can't say that it's perfect, but I think it should be usable. * Started to support multiple nzb files passed on the commandline. So far, it's been pretty straightforward. * Added --dlpath commandline option. You can use this option to specify a single path into which all files should be downloaded and decoded. * Added --dlrelative commandline option. This option tells nzbperl to decode files into the same dir that each originating nzb file was found. This should allow people to set up a directory structure for multiple files and have all things downloaded into nice little organized paths. Some additional explanation: Say you have 3 nzb files: /path/to/foo.nzb /path/to/bar.nzb /path/to/fubar.nzb Where each nzb file contains (for this example), 2 files. If --dlrelative was given on the commandline, the output dirs might contain: /path/to/foo.nzb /path/to/foo1.file /path/to/foo2.file /path/to/bar.nzb /path/to/bar1.file /path/to/bar2.file /path/to/fubar.nzb /path/to/fubar1.file /path/to/fubar2.file Ok, I've done a horrible job of explaining this. :) Basically, if you want files to be downloaded to and decoded to the same dirctory that their originating nzb file came from, use this option. I should also note that --dlpath and --dlrelative are conflicting options, (that generate an error) even in the case where a single nzb file is in the current directory. -------------------------------------------------------------------------- 2005-04-07 - * Fixed bug in uudeview changes finding from 3/27 -------------------------------------------------------------------------- 2005-04-02 * Fixed two divide by zero errors that would happen when a horribly mangled nzb file is encountered (ie. one where the adjusted avg is zero, like if a file only contains segment number 2). -------------------------------------------------------------------------- 2005-03-27 - Started work on next version * Incorporated Frederic's patch to prevent server hostname from being DNS cached for multiple connections. Previous versions would always end up opening multiple connections to the same server (IP), even if the news provider used DNS for load balancing. Thanks man! * Incorporated Frederic's suggestion to check for uudeview existence before going off and downloading a bunch of stuff. nzbperl will now crap out if it can't find uudeview. * Added a new --uudeview option so that uudeview doesn't *have* to be in the path anymore. Same thing works in the config file. If the file doesn't really exist, a warning is shown and the path is still checked. -------------------------------------------------------------------------- 2005-03-20 - * Fixed a bug in the login/reconnect logic that caused a crash during a login after a reconnect. * No longer use the autoclear (-c) option to uudeview, this just made things more complicated then they needed to be (backwards logic). Cleanup is done by hand now. * There are now 3 separate keep options. The original --keep was promoted to --keepparts. This was a frequently requested feature. Here's the breakdown: --keepparts : Always keep temp .parts files on disk after decoding. --keepbroken : Keep temp .parts files on disk for those files that failed to decode successfully. This should be thought of as "keep downloading broken files and only leave encoded .parts files on disk for broken files" --keepbrokenbin : Allow uudeview to generate broken binary files when a file is corrupt. This should be thought of as "keep downloading broken files (missing segments etc) and decode to broken binary files but do not leave the encoded .parts files on disk". For my own sanity and perhaps clarity for users, here's a matrix: | continue | | | | | dloading | keep all | keep bad | decode to broken | | broken | .parts | .parts | binary files | +---------------+----------+----------+----------+------------------+ | keepparts | no | yes | yes | no | | keepbroken | yes | no | yes | no | | keepbrokenbin | yes | no | no | yes | +---------------+----------+----------+----------+------------------+ Previously, the --keep option was doing double or triple duty. It's maybe more complicated now, but there's a more granular level of control that folks wanted. In hindsight, --keepparts is probably way less useful now. -------------------------------------------------------------------------- 2005-03-19 - * Started reworking the --keep option to make people happy. Will finish tomorrow. * Added config file support! Options can be placed into the config file and will be read prior to other commandline options (except for the new --config option). Once a config file has been set up, starting nzbperl is now as easy as nzbperl.pl , which is a good thing. I've also done it in a way that allows for future commandline args to be handled via the config file without additional code. * Fixed a bug that caused status messages to show up on the help screen. -------------------------------------------------------------------------- 2005-03-17 - * Changed a seriously horrific bug which caused a premature end of segment in the NNTP code. The double dot escape crap was happening before the end of message check was happening, so if there was a double dot at the beginning of any line in the segment (which I guess happens rather frequently in yenc files), it would trigger a premature end of segment and possibly allow mangled crap to sneak into the display and basically bungle things up. The escaping only happens now right before the line is sent to the output file. Sheesh. -------------------------------------------------------------------------- 2005-03-12 - * Fixed a bug in the nzb parsing routine. Previous versions assumed that the segment elements were listed in the correct order, which apparently isn't always the case (at least 1 user has nzb files that have segments in backward or otherwise different order). The "number" attribute is now used as the segment index, so this should fix some problems (primarily related to the sanity checking algorithm). -------------------------------------------------------------------------- 2005-02-16 - * Changed the regex filtering to happen *before* the skip. I think it was less useful the other way 'round. -------------------------------------------------------------------------- 2005-02-06 - * Fixed a bug that mucked up the display if the user hit the enter key * Started building server-initiated disconnect detection and reconnect logic (after a delay), but it's going to take a while to get this right. * Program shouldn't use 100% CPU when the server disconnects now. At least that's a step in the right direction. * Skip the socket closing steps at disconnect time if the socket is already closed. * Added --retrywait commandline option to control how long to wait between reconnect attempts (default = 60 seconds). I wonder if there should be an absolutel minimum here...we don't want people setting this too low and beating up on servers... * If the ETA is greater than 240 hours (10 days), we now just punt and give a ??:??:?? estimate. I suspect that anything over 10 days can't really be accurate anyway. :) This just helps to keep the screen a bit cleaner. * I've discovered at least one buggy NNTP server implementation that I think breaks spec (doesn't stick to the RFC). I suppose this is friggin commonplace. Anyway, there are apparently cases where a server will provide more than one response to a BODY command. This is not good, and it makes it a pain in the ass to handle. The specific case I'm seeing is a 222 (body retieved) followed by a 502 (access restriction). Wonderful. * Ok, I *think* I now handle the above case (5xx following a 2xx). At least the server message (reason) is displayed. * Regular expression filtering on subjects in the NZB file is now supported with the --filter option. This can be useful if you have several things in one NZB and you just want to process one subset of the files. I should point out too, that regex filtering is done *after* any --skip files have been discarded. That is, the regex filtering filters what's left after a --skip. -------------------------------------------------------------------------- 2005-01-27 - 0.6.1 likely released tonight... * Well, several people have discovered that I screwed up the redirection on the call to uudeview...which can ultimately screw up downloads badly. Sorry users! Still not sure why this never failed on my setup, but oh well. Omen submitted a patch for this, and I incorporated it tonight. * Incorporated a modified version of Omen's patch to clamp the output of the hrsv function to an integer (to prevent long precision floating point output from mucking things up). -------------------------------------------------------------------------- 2005-01-24 - * Changed decoding failure message wording to emphasize FAILED. :) * Change the uudeview command to append to the decode debug file instead of overwriting it each time (should help to facilitate debugging of mangled download files). * Remove the decode debug file on successful program startup. -------------------------------------------------------------------------- 2005-01-23 - Continued 0.6 development/bugfixes... * Added --skip commandline option that will skip over the first files in the nzb. Useful for restarting large downloads without having to let nzbperl detect/discover filenames. In other words, if you aborted a session after 20 of 25 files were downloaded, running later with --skip 20 would only stick the remaining 5 files from the nzb into the download queue. -------------------------------------------------------------------------- 2005-01-22 - Continued 0.6 development/bugfixes... * Total file count and remaining count are now decremented when a file is skipped because it already exists on disk. * Total job size in bytes is now decremented appropriately when a file is skipped because it already existed on disk. Total bytes downloaded is also adjusted to account for the skip. I hope I managed to get this right. :) * Improved failed segment handling so that the totals are adjusted to reflect that the file was dropped (when not using --keep) -------------------------------------------------------------------------- 2005-01-20 - Continued 0.6 development/bugfixes... * Incorporated Omen's suggestion to use -e instead of using a whole subroutine to do file existence checking. * Implemented modified version of Omen's file download size spacing patch to reduce the jump around text effect. * I thought I had this fixed earlier, but I screwed up. It's hopefully now impossible to bump the bandwidth setting below 1kBps. * Incorporated Omen's ETA smoothing patch so that the ETA is based on a set of average speeds and tends to be less jumpy. * Incorporated Omen's patch for decoding display clean up. Long filenames should no longer mangle the screen, and it's generally prettier. * Omen was a huge help! Props. -------------------------------------------------------------------------- 2005-01-16 - Continued 0.6 development/bugfixes... * Added ability to log status messages via --log commandline arg This is especially useful for debugging/troubleshooting problems. * Refactored the commandline handling stuff into a function. It was getting crowded up there. * NZB parsing now grabs the posting date from the xml attributes * Implemented --insane command line option to instruct nzbperl to bypass all NZB sanity checks. If this is given, all files (even those with suspect/broken/bad parts) will be attempted. * Implemented --dropbad command line option. If this is given, all files that have one or more suspect parts will be dropped from the NZB file list and will NOT even be attempted. This should probably be turned on by default, but I'm only 95% confident in the bad part detection heuristic, so the default behavior is to prompt the user.. * Implemented mini menu to prompt the user when there are bad file parts detected. It can allow the user to quit out, review the bad parts details (ie. what parts of what files), and/or set the --dropbad or --insane options. Hopefully this should help people understand when they have a broken or bad nzb file. -------------------------------------------------------------------------- 2005-01-13 - Continued 0.6 development/bugfixes... * Verified a few pretty serious problems the other night: - NNTP session timeouts aren't handled - A bad nzbfile (where many segments are suspect [small size]) can still result in a large/long download and bandwidth use. So...I'll be looking into addressing these as part of 0.6. * Started building nzb sanity checker. It will look at the average size of all parts in each file, excluding the last part and will then locate parts that have a size difference over a given % threshold (currently 25%). If there are busted files (and other commandline options haven't been specified), a little mini menu will be used to determine how to proceed (not yet built)... -------------------------------------------------------------------------- 2005-01-10 - Continued 0.6 development/bugfixes... * I think I've identified the culprit for the broken downloads that left parts files on disk... * Return code from uudeview is now captured and used to detect success or failure in decoding. * On decode failure, a message is shown in the status screen. If the keep setting was specified, the file is left on disk, otherwise it's deleted. * An alternate port number can now be given, attached to the --server commandline option in the form --server [:port] Like --server your.news.host:80 The same syntax should also work with the NNTPSERVER environment variable, but it's probably nonstandard. Some news providers allow access on alternate ports to prevent firewall issues etc....so now nzbperl will work with these. * Now properly detects when a BODY request to the news server fails. Previously, the code was only looking for positive responses. A status message and the server response code/message are shown when this happens. * Depending on the --keep option, a BODY segment fetch failure will either skip the remaining segments or will just skip to the next file (after cleaning up nicely) * Totals are now modifed on the fly to reflect segments that can't be fetched from the server (bumped down) * No longer allows bumping speed up from max/unlimited speed (there was a bug that set it to 1kBps when bumped up from unlimited -- doh) :) -------------------------------------------------------------------------- 2005-01-09 - Troubleshooting, started 0.6 * Started troubleshooting bug with 0.5 that left broken *.parts files on disk. Believed to be caused be broken nzb files (missing segments) and/or segments missing from the server. -------------------------------------------------------------------------- 2005-01-03 - Finishing up 0.5 version for release * Added '+' and '-' minus keys to manually increase and decrease the target bandwidth by 1kBps (respectively) * Fixed a bug related to weird/bad characters in "true" filenames causing a regex to fail and the app to abort. True filenames are escaped now for special characters before looking for files. (thanks to Sascha from Germany for reporting this one!) * Releasing tonight. -------------------------------------------------------------------------- 2005-01-02 - Major rehaul release (version 0.5) almost complete.. Just continued development and polishing things up for the release. Some continued changes: * 'h' or '?' key during operation now shows a help screen. Downloads even continue while the help screen is being displayed. * Added status display. A small "window" of sorts that scrolls informative messages with timestamps. * Unified version number (via global) throughout the code, and version bumped all the way up to 0.5 (for the helluvit) * Rebuilt skipping when file already exists on disk. * Fixed bug related to pending socket data during skip. -------------------------------------------------------------------------- 2005-01-01 - Major revamp underway After starting the revamp several times, I finally jumped in feet first and started building the thing from the ground up. I was flustered with my options -- multiplexing via Net::NNTP is likely impossible, and the [new] threading model in Perl is just a piece of shit...both of these block created the new monster: a revamped nzbperl that handles multiple server connections with a single thread. I like this. The code base is significantly different, but here are the real functional changes worth taking note of: * Multiple server connections. Control the number with --conn param. Default value is 2. Multiple connections are accomplished by multiplexing sockets with select(). Each connection services a single file (so large files with many segments are still capped at the per-connection speed). In general, though, users with per-connection bandwidth limits should be able to achieve much higher throughput. * No more dependency on Net::NNTP. The tied filehandle thing alwyas seemed like a hack, so I ended up building my own hack into nzbperl to support muliplexed sockets (see above item) * Much improved display, including colors. Status for each individual connection is shown. Better totals, including session thruput, files remaining, and estimated time remaining (ETA). * New dependency on Term::Cap (which you probably already have) for screen positioning. * New dependency on Term::ANSIColor if you want color output (it's really pretty, you should try it!) :) The screen even looks pretty damn decent in 80-column text consoles too. If you don't want color, there's a --nocolor option to turn it off. -------------------------------------------------------------------------- 2004-11-06 - Version 0.03 released Implemented 'q' key to force a clean quit. Implemented 's' key to skip the current file download. nzbperl.pl is now able to auto skip files that it has already downloaded from an nzb file! This is a pretty huge step forward for this program (and one of the most requested features). The behavior may need some explaining though. One of the flaws in the usenet binary system is that subject lines cannot (or currently *do* not) provide deterministic or accurate filenames. In order for an NNTP tool to determine that it has already downloaded a file on a previous run, it must either store a history on the filesystem somewhere, or it can download the first part of a file segment to determine the true filename. I decided to *not* implement an .rc file or other state-tracking file...at least not for now. After nzbperl grabs the first segment of any file, it will use the yenc or uuencode header information to determine if the file already exists on disk (in the current working directory). If it does, the rest of the file will not be downloaded. To force nzbperl to download files that already exist in the current directory, you can pass the --redo command line argument. The end result is that nzbperl can now skip files that it has already downloaded. The down side is that it still has to fetch the first part of the file in order to determine if the file has already been processed. Overall, it should still be faster and more convenient than grabbing everything over again or hand-editing the nzb xml. -------------------------------------------------------------------------- 2004-11-03 - Added filesize checks in the buildDispLine routine. Apparently there were cases where a zero byte filesize was being used and a silly divide by zero was kicking in. Although I'm not sure why some people were having a zero byte file size passed in, I check for it anyway now. Now uses NNTPSERVER environment variable in addition to command line arg. Command line arg takes priority if specified. -------------------------------------------------------------------------- 2004-11-03 - Changelog started. -------------------------------------------------------------------------- - Changed shell call from uudecode to uudeview