Clomps conf: Difference between revisions

From Lundman Wiki
mNo edit summary
mNo edit summary
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
FXP.One is a whole new system to FTP and FXP. It is not just another
# Which FXP.One to connect to. Omit PASS to have app prompt for passwd.
[[FXP.One:Clients|FTP client]]. It is in fact a very flexible FTP/FXP engine. This engine does
FXPONE|HOST=server.that.runs.fxpone.com|PORT=8885|USER=admin|PASS=admin|ssl=forced
all the hard work with dealing with FTP sites. Built into that is
a very simple but powerful API protocol. The idea is then, if someone
wants to do an FTP, or FXP, client they can then make one without
the FTP hassles. Currently there already are multiple [[FXP.One:Clients|clients]]. They
all talk to the FXP.One engine, and you can interchange the [[FXP.One:Clients|clients]].


That is, use one client to create and queue up some items and start the
Define where to connect for the FXP.One engine. It can be the same server as clomps run on, use localhost. If you leave the '''PASS=''' out, then clomps will prompt for passwords.
queue process. At a later time, a different client, from a different location
'''ssl=forced''' on this line refers to the connection to the engine. Encryption is a good idea, except if you are using localhost, and nobody else has root on your server.
can connect and check on the progress of that queue, change it, add to it and
so on.
# Define sites and paths.
<paypal></paypal>
SITE|NAME=ASite1|DIR=/x264-tv/|USESKIP=1|INCTEST=%s-(INCOMPLETE)|NUKETEST=NUKED-%s|TIMEFILE=.asite.timestamp
SITE|NAME=Another|DIR=/tv-x264/|USESKIP=1|INCTEST=%s-(INCOMPLETE)|NUKETEST=NUKED-%s|HIDE=American.Idol*
SITE|NAME=Another|HIDE=Additional*/*HideStrings*
SITE|NAME=glftpdSite|DIR=/tv-x264/|USESKIP=1|INCTEST=(INCOMPLETE)-%s|NUKETEST=!NUKED-%s
SITE|NAME=home|DIR=/tv/|DIR=/drive2/tv/|USESKIP=1|INCTEST=%s-(INCOMPLETE)|NUKETEST=NUKED-%s


Here we define the sites to connect to, and which directories to list. Note that '''home''' has two directories listed, they are combined.
TIMEFILE=filename can specify the timestamp file, if left out, clomps will make it based on the config filename + .timestamp


'''FXP.One V2 released! Now comes with HTML5 WebSocket FTP Client! IRC Auto-trader bot! '''
If you run this clomps.conf now, you will get an output like:


  The FXP.One engine features:
  ./clomps -vf example.conf
  * Full FTP and FXP capabilities.
  * SSL/TLS support, auto-sensing and forced.
connected to FXP.One
  * SSL/TLS data support, auto-sensing and forced.
  successfully negotiated SSL
  * SSCN secure data FXP support.
Successfully connected to FXP.One
  * CCSN secure data FXP support.
  Site glftpdSite processing dir '/tv-x264/'
  * XDUPE aware for faster queue processing.
Site home processing dir '/tv/'
  * Auto resume, or overwrite options
  Site Another processing dir '/tv-x264/'
  * Resume last (re-queue all resume items last for faster queue processing)
  Site ASite1 processing dir '/x264-tv/'
  * FXP direction control (if one site is firewalled)
  Site home processing dir '/drive2/tv/'
  * PRET Pre-transfer support for ring-sites.
      Another finished processing: 111 files (  1/  4)
  * Skip lists for both files and directories
          home finished processing: 3135 files (  2/ 4)
  * Pass lists for both files and directories (opposite to skip list)
    glftpdSite finished processing: 158 files ( 3/  4)
  * Move-first for both files and directories
        ASite1 finished processing: 166 files ( 4/  4)
  * Automatic skip of empty files and directories.
   
* Encodes all file and directory names as to handle any locale.
-----======> DISPLAY MATRIX <======-----
T |ASite|Anoth|glftp|home | New Items
--+-----+-----+-----+-----+-----------
d |INC  |MISS |MISS |MISS | Best.Laid.Plans.2012.1080p.BluRay.X264-7SinS
d |INC  |MISS |MISS |MISS | To.Catch.A.Thief.1955.1080p.BluRay.x264-HD4U
  d |    |MISS |MISS |MISS | The.Summer.Of.Massacre.2011.720p.BluRay.x264-UNTOUCH
  ABLES
  d |    |MISS |NUKE |MISS | Under.Still.Waters.2008.720p.BluRay.x264-SWAGGERHD
   
Which shows new releases, by showing up (names on the right side) they exist somewhere. Any site column with a blank entry, means the release is there and complete. '''INC''' means that the release is incomplete (INCTEST patten matches), and similarly '''NUKE''' means nuked. '''MISS''' means this release is not on this site.




=== Automatic queueing ===


[[Image:ufxp.screen1.jpg|right|thumb|100px|UFxp]]
In addition to finding new entries, you can ask clomps to create queues. The queue lines are defines with '''pass number'''. Starting from 1. This means, all queue lines with pass=1 will be started at the same time. Once all pass=1 transfers are complete, clomps will run pass=2. This allows for all-concurrent queueing, or, sequentially queueing.  


For example (Spaces added between '|' separator for readability):


==Getting Sources==
AUTOQ|PASSNUM=1|FROM=glftpdSite|TO=ASite1 | ACCEPT=the.simpson*/family.guy*/american.dad*/How?I?met?your?mother* | REJECT=*FINSUB*/*AC3* | INCSKIP
CONTINUED|ACCEPT=House*/firefly*
CONTINUED|ACCEPT=orphan.black*|REJECT=*webrip*


Latest tarball.
Which would automatically transfer items from '''glftpdSite''' "/tv-x264/", to '''ASite1''' "/x264-tv/". But only releases matching "the.simpson*" etc, unless they are *FINSUB* etc. The '''INCSKIP''' tells clomps not to transfer things that are incomplete. Clomps will then only transfer completed-releases. If you remove '''INCSKIP''' clomps will start to move incomplete releases as well.
[http://www.lundman.net/ftp/fxp-one-2.0.tar.gz FXP.One-2.0.tar.gz]
                         
Older versions:
[http://www.lundman.net/ftp/fxp-one-1.9.tar.gz FXP.One-1.9.tar.gz]
[http://www.lundman.net/ftp/fxp-one-1.7.tar.gz FXP.One-1.7.tar.gz]
[http://www.lundman.net/ftp/fxp-one-1.6.tar.gz FXP.One-1.6.tar.gz]
[http://www.lundman.net/ftp/fxp-one-1.4.tar.gz FXP.One-1.4.tar.gz]
[http://www.lundman.net/ftp/fxp-one-1.3.tar.gz FXP.One-1.3.tar.gz]
[http://www.lundman.net/ftp/fxp-one-1.2.tar.gz FXP.One-1.2.tar.gz]
[http://www.lundman.net/ftp/FXP.One.v1.1.b737.tgz FXP.One.v1.1.b737.tgz]
[http://www.lundman.net/ftp/FXP.One.v1.1.b735.tgz FXP.One.v1.1.b735.tgz]
[http://www.lundman.net/ftp/FXP.One-v1.0.b598.tgz FXP.One-v1.0.b598.tgz]


The right-hand side in ACCEPT and REJECT statements are slash-separated list of matches. Ie, "*simpsons*/blue.planet*" would match on "*simpsons*" and "blue.planet*", with glob-style pattern matching. This means "I.hate.simpsons" would match (in the middle) but "the.blue.planet" would not (since it was not "*blue.planet*". No starting "*"). Note that this would also fail on "Blue_Planet" and "blue planet", since the period "." is literal. The best way is to use "blue?planet*".  All matches are '''case-insensitive'''.


CONTINUED lines, can add to the ACCEPT and REJECT lines if placed after an AUTOQ command. It is for readability and flexibility.


[[CVS]] Use CVS for cutting-edge version.


[[Compile]] FXP.One
AUTOQ|PASSNUM=2|FROM=ASite1|TO=home | ACCEPT=* | REJECT=*dvdrip*


[[FXP.Oned_Changelog|Changelog]]
After pass=1 has completed, move '''everything''' from '''ASite1''' to home. Except for dvdrips. Incompletes will also be started.
 
[http://www.lundman.net/unix/FXP.One_commitlog.txt Old CVS Changelog]
 
==Clients==
 
[[FXP.One:Clients|Clients]]
 
 
 
==General tech specs==
 
[[Error Codes]] For Protocol error codes and definitions.
 
[[YNA Type]] Yes, No and Auto type
 
[[Encoding]] The directory and file encoding scheme used.
 
[[Site Definition]] Current SITE definition and its members.
 
[[Extra Site]] members also understood by the engine.
 
[[Client Site]] recommendations.
 
[[Phases]] required to do a file transfer. (need to do directory listing some day)
 
==API COMMANDS==
 
[[Protocol]] definition and specifications.
 
 
 
== Local transfers ==
 
[[FXPOne:local|local]]
 
 
 
== Help ==
 
[[FXPOne|FAQ]]

Latest revision as of 07:18, 14 May 2014

# Which FXP.One to connect to. Omit PASS to have app prompt for passwd.
FXPONE|HOST=server.that.runs.fxpone.com|PORT=8885|USER=admin|PASS=admin|ssl=forced

Define where to connect for the FXP.One engine. It can be the same server as clomps run on, use localhost. If you leave the PASS= out, then clomps will prompt for passwords. ssl=forced on this line refers to the connection to the engine. Encryption is a good idea, except if you are using localhost, and nobody else has root on your server. 

# Define sites and paths.
SITE|NAME=ASite1|DIR=/x264-tv/|USESKIP=1|INCTEST=%s-(INCOMPLETE)|NUKETEST=NUKED-%s|TIMEFILE=.asite.timestamp
SITE|NAME=Another|DIR=/tv-x264/|USESKIP=1|INCTEST=%s-(INCOMPLETE)|NUKETEST=NUKED-%s|HIDE=American.Idol*
SITE|NAME=Another|HIDE=Additional*/*HideStrings*
SITE|NAME=glftpdSite|DIR=/tv-x264/|USESKIP=1|INCTEST=(INCOMPLETE)-%s|NUKETEST=!NUKED-%s
SITE|NAME=home|DIR=/tv/|DIR=/drive2/tv/|USESKIP=1|INCTEST=%s-(INCOMPLETE)|NUKETEST=NUKED-%s

Here we define the sites to connect to, and which directories to list. Note that home has two directories listed, they are combined. TIMEFILE=filename can specify the timestamp file, if left out, clomps will make it based on the config filename + .timestamp

If you run this clomps.conf now, you will get an output like: 

./clomps -vf example.conf

connected to FXP.One
successfully negotiated SSL
Successfully connected to FXP.One
Site glftpdSite processing dir '/tv-x264/'
Site home processing dir '/tv/'
Site Another processing dir '/tv-x264/'
Site ASite1 processing dir '/x264-tv/'
Site home processing dir '/drive2/tv/'
      Another finished processing:  111 files (  1/  4)
         home finished processing: 3135 files (  2/  4)
   glftpdSite finished processing:  158 files (  3/  4)
       ASite1 finished processing:  166 files (  4/  4)


-----======> DISPLAY MATRIX <======-----

T |ASite|Anoth|glftp|home | New Items
--+-----+-----+-----+-----+-----------
d |INC  |MISS |MISS |MISS | Best.Laid.Plans.2012.1080p.BluRay.X264-7SinS
d |INC  |MISS |MISS |MISS | To.Catch.A.Thief.1955.1080p.BluRay.x264-HD4U
d |     |MISS |MISS |MISS | The.Summer.Of.Massacre.2011.720p.BluRay.x264-UNTOUCH
ABLES
d |     |MISS |NUKE |MISS | Under.Still.Waters.2008.720p.BluRay.x264-SWAGGERHD

Which shows new releases, by showing up (names on the right side) they exist somewhere. Any site column with a blank entry, means the release is there and complete. INC means that the release is incomplete (INCTEST patten matches), and similarly NUKE means nuked. MISS means this release is not on this site.


Automatic queueing

In addition to finding new entries, you can ask clomps to create queues. The queue lines are defines with pass number. Starting from 1. This means, all queue lines with pass=1 will be started at the same time. Once all pass=1 transfers are complete, clomps will run pass=2. This allows for all-concurrent queueing, or, sequentially queueing.

For example (Spaces added between '|' separator for readability): 

AUTOQ|PASSNUM=1|FROM=glftpdSite|TO=ASite1 | ACCEPT=the.simpson*/family.guy*/american.dad*/How?I?met?your?mother* | REJECT=*FINSUB*/*AC3* | INCSKIP
CONTINUED|ACCEPT=House*/firefly*
CONTINUED|ACCEPT=orphan.black*|REJECT=*webrip*

Which would automatically transfer items from glftpdSite "/tv-x264/", to ASite1 "/x264-tv/". But only releases matching "the.simpson*" etc, unless they are *FINSUB* etc. The INCSKIP tells clomps not to transfer things that are incomplete. Clomps will then only transfer completed-releases. If you remove INCSKIP clomps will start to move incomplete releases as well.

The right-hand side in ACCEPT and REJECT statements are slash-separated list of matches. Ie, "*simpsons*/blue.planet*" would match on "*simpsons*" and "blue.planet*", with glob-style pattern matching. This means "I.hate.simpsons" would match (in the middle) but "the.blue.planet" would not (since it was not "*blue.planet*". No starting "*"). Note that this would also fail on "Blue_Planet" and "blue planet", since the period "." is literal. The best way is to use "blue?planet*". All matches are case-insensitive.

CONTINUED lines, can add to the ACCEPT and REJECT lines if placed after an AUTOQ command. It is for readability and flexibility. 


AUTOQ|PASSNUM=2|FROM=ASite1|TO=home | ACCEPT=* | REJECT=*dvdrip*

After pass=1 has completed, move everything from ASite1 to home. Except for dvdrips. Incompletes will also be started.

Retrieved from "http://lundman.net/wiki/index.php?title=Clomps_conf&oldid=4529"