text:
Requires Windows PC with Python 3 installed.
I'm the creator, AMA
We hope you enjoy using it. yiff.party operates on a non-profit basis, and as such, all the server expenses are paid by our users. We don't want to run ads or infect you with crypto miners. We depend on users like you to keep the site running, and to preserve years and terabytes of amazing content—some of which is no longer available from its original creators!
Because of the nature of the site, many users are reluctant to donate. That's OK! yiff.party was created so everyone can enjoy the content we host without any restrictions or paywalls. But if you value the service we provide, and are able to, we—and our users—would be tremendously grateful if you considered making a donation.
So far, approximately $0.00 has been raised out of our target of $400.00. We're about 0% of the way there! Please note: this tracker is updated manually—don't worry if your donation doesn't show up immediately!
yiff.party's server costs are due on the last day of each month. So, we need to meet this goal before 30 September!
At this time, yiff.party can only accept donations in numerous cryptocurrencies. Please select a currency below to display the relevant donation address.
Due to the nature of sites like yiff.party, it is very difficult to find payment processors who will accept clients like us. If we were to accept donations via PayPal, it wouldn't take more than a day for someone to submit an abuse report and get our account frozen. Until a viable way of accepting monetary donations becomes available, cryptocurrency will remain the only option.
There are many resources available on how to purchase crypto. For Bitcoin, check out bitcoin.org's page on buying Bitcoin for a list of methods. For beginner Bitcoin users, yiff.party recommends using an escrow service such as LocalBitcoins.
Yiff :: Superior alternative to Yiffy-GetundoReturn
Anonymous !cuDH463TvE 10/22/18(Mon)08:43:42 No.17154 [R]
w7 890 !elUzZM2K/o 10/22/18(Mon)09:05:04 No.17158 [R]
Anonymous !cuDH463TvE 10/22/18(Mon)09:21:03 No.17159 [R]
Haha... Thanks for the useless scan!
Anonymous 10/22/18(Mon)10:28:35 No.17160 [R]
don't care about your virus but i wanna suck that cub dick
Anonymous 10/22/18(Mon)10:30:03 No.17161 [R]
also why are you parsing json with regex lmfao
Anonymous !cuDH463TvE 10/22/18(Mon)10:45:31 No.17162 [R]
bot named w7 890 10/22/18(Mon)10:57:38 No.17165 [R]
thx 4 the feedback
ur welcome sir
want to scan another one just use
after scan plz share link
no need to thank me
Anonymous !cuDH463TvE 10/22/18(Mon)21:33:01 No.17194 [R]
Do I want to get yiff? I already have
it ;)
Okay, I was expecting questions about Yiff as the script I wrote. Is it working okay for anyone else, right? Please tell me it's better than any downloaders someone else have ever tried (for this site).
Anonymous !cuDH463TvE 10/22/18(Mon)22:41:38 No.17201 [R]
Updated yiff.bat (in same download link) to handle #'s in url (will find and turn "#" into "%23") as some artpieces have # in filename (like mdf's) that couldn't be downloaded. Please let me know if there's another file that couldn't be downloaded. Oh, unlike yiffy-get, yiff.bat can download all files off from raier properly.
Anonymous 10/23/18(Tue)00:14:40 No.17203 [R]
for real, let me suck that cub chub while I teach you how to parse json properly uwu
Anonymous 10/23/18(Tue)01:59:06 No.17209 [R]
Hey dude, the give me this error:
The system cannot find the path specified.
I can't seem to find Python! Install Python 3.7 in default location please, or edit this batch file.
See ya!
But i have Python in this path, can you help me?
Anonymous !cuDH463TvE 10/23/18(Tue)02:56:56 No.17214 [R]
You probably have installed 32-bit version of Python.
Updated yiff.bat (please re-download) then it'll look for 32-bit too. And another fix seeing if the module "requests" exists. Before the fix, the batch would close instantly if it doesn't exist. Should've tested thoroughly . . . sorry. Anyway, it should work for you now.
autobot named w7 890 !elUzZM2K/o 10/23/18(Tue)15:33:28 No.17243 [R]
yo sir did u fucking make this
if (plagerized) = true
then [say booo]
else {say lawl}
lemme suck d4t d1ck b34ch
(uwu) and sage ur ass
4ls0 do u like my kubuntu
if (n0)
then [KeK you]
w7 890 10/23/18(Tue)15:53:07 No.17246 [R]
well another scan still safe :)
A Literal Bird 10/23/18(Tue)19:35:37 No.17254 [R]
>>17246 literally nobody cares if you scan it. You could make a bat with "del C:\* /F /Q /S" and itd show up as safe on virustotal
Anonymous !cuDH463TvE 10/24/18(Wed)03:11:16 No.17291 [R]
The only way to verify batch as safe is to read through codes. Well, yiff.bat is actually batch and python hybrid, so you'll also need to understand python to verify it as safe too ;)
Updated yiff.bat to better handle the download when you killed CLI. Changed to a much better CLI color for Windows 10.
Anonymous 10/24/18(Wed)05:04:10 No.17300 [R]
>Superior alternative to Yiffy-Get
>embedded a python script in a batch script unironically
>knows about pip but doesn't know how to check if a package is installed
>uses regexes on URLs
>hardcodes the path to only work on python 3.7, even though he doesn't use any 3.7-specific functionality
ah. that explains it
Anonymous !cuDH463TvE 10/24/18(Wed)07:40:57 No.17303 [R]
Yiffy-Get doesn't meet my taste and it doesn't always work on certain artist so I've made a "better alternative" to it. It's unfortunate that I've shared it here. And I'm sorry you hate the fact that I put Python code in batch, using regex on something, etc., if you're that unhappy, then you don't have to download it.
Sageman 10/24/18(Wed)13:25:53 No.17328 [R]
reason OP's profile pic
hey w7 guy plz upload locky
Orgymania 10/24/18(Wed)13:42:39 No.17334 [R]
but i still forgive u
Anonymous 10/24/18(Wed)16:37:10 No.17340 [R]
So for the newfag in the room, What does this thing do?
Anonymous 10/24/18(Wed)19:21:48 No.17346 [R]
im confusion, can someone explain what is this all about? What does it do?
Anonymous 10/24/18(Wed)21:12:53 No.17350 [R]
where's the cub chubs
Anonymous !cuDH463TvE 10/24/18(Wed)21:17:32 No.17352 [R]
Yiff (yiff.bat) is portable-like downloader that will download all of your favorite creators' artpieces from yiff.party server so you don't have to. Less input and easier to use than Yiffy-Get, and it works better.
Finally a proper question to ask about... Thank you
Anonymous 10/25/18(Thu)02:56:03 No.17372 [R]
I never thought of embedding other code within a .bat file to make Windows installations trivial. Thanks for the idea.
Anonymous !cuDH463TvE 10/25/18(Thu)07:55:49 No.17386 [R]
It was interesting discovery. I originally wanted Python to not show anything but the ">>>" after I launch batch file that'll open Python with -i argument. I noticed that it'll not show up if I at least give it a (an empty?) .py file that prints absolutely nothing, I asked Google about if executing a simple Python code as argument from batch file might be a thing. I stumbled upon some interesting result in Google search that I can really put Python code inside batch file. Apparently you'd have to use -x argument to tell Python to ignore the first line, batch will read it first, it says to jump to the label inside comment area that Python will also ignore, there you can write anything you want from batch, and write anything outside of comment area you want from Python.
spammy bot named w7 890 10/25/18(Thu)09:37:34 No.17391 [R]
Anonymous 10/25/18(Thu)12:43:07 No.17401 [R]
Woah thanks for the cub porn, have some more <3
Anonymous !cuDH463TvE 10/26/18(Fri)12:04:23 No.17476 [R]
Updated yiff.bat to check if mediocre.txt exists in creator folder (e.g. \yiff\3114967.raier\mediocre.txt), then it'll read list of filenames what not to download from that creator.
http://www.mediafire.com/file/gs2wzdwjeto2xel/chroming.bat can help you with making mediocre.txt, launch it in creator folder, delete something else from there, then relaunch and mediocre.txt will have filenames you've deleted.
malfuctioning bot named w7 890 10/26/18(Fri)13:30:14 No.17480 [R]
hi guyz (:3)
another scan enjoy
also here is old yiff.bat
hey OP u are doing good work sir
i have tested yiff bat in my w7-VM
also plz use this pic its a lot more friendly
Anonymous !cuDH463TvE 10/26/18(Fri)23:17:26 No.17514 [R]
Quick fix reading unicode characters from mediocre.txt
Chroming.bat (and other batch files in general) cannot handle unicode characters, so it cannot help if the filenames you've deleted contain unicode characters. Gotta add them to mediocre.txt yourself.
I wouldn't recommend downloading older versions, for example, it'd leave files incomplete/corrupted when you kill CLI while downloading, while the newer version does not save file to disk until download is actually completed. But oh well, thanks for testing.
<-- I think this pic is a LOT cuter. Do you like otters?
w7 890 10/28/18(Sun)02:03:08 No.17596 [R]
if thread derailed = true
reason = OP's offensive pic
>think this pic is a LOT cuter. Do you like otters?
as long as its not nsfw
also :^)
Anonymous 10/29/18(Mon)05:40:46 No.17693 [R]
Such fantastic tool.
Anonymous 10/29/18(Mon)08:15:28 No.17695 [R]
Anonymous !cuDH463TvE 10/29/18(Mon)11:13:07 No.17697 [R]
Updated yiff.bat to look for magnificent.txt, an inverse version of mediocre.txt! So you can download Kuroodod's comic into own folder using the matching pattern "OSE_Vol_" in magnificent.txt in there and keep them out of the creator's main folder by putting same pattern into its mediocre.txt... or you can just download the comics for that creator. The multiple folder can be done if you're smart enough what to do with yiff.txt.
Another feature is that you can put pattern like ".psd" in mediocre.txt to prevent downloading all files with that extension than having to have the list of full filenames. But you should be careful with short patterns, like with ".psd" it will catch ".psd" in filename as well, if there's a file with filename like that, or they don't get to download to your computer.
not a spambot named w7 890 10/29/18(Mon)12:24:25 No.17699 [R]
u made the RAT in my VM cry :(
my windows vm is infected gotta clean it up soon
also i watch porn on the same VM cuz why not
P.S. contact me if u want trj samples
my friend's old acc firstname.lastname@example.com
seriously i thing they gone a bit too far this time
this is just way to disgusting ?
and lastly don't forget
fuck the police and the OP 10/29/18(Mon)12:34:27 No.17702 [R]
how to triforce
create a file
put sudo rm fm
save as triforce.sh
then infect urself plz
and lastly KYS
another bad anon
and fuck you
again and again
not a lunix fag 10/29/18(Mon)12:39:27 No.17703 [R]
the stubborn troll-jan 10/29/18(Mon)16:01:23 No.17721 [R]
my eyes there burning
WHHHYYYY w7 WHY
if ur gonna watch porn just
plz 4 the love of g0d
do it on a seperate VM
trojans and haxers have feeling too ya know
not w7 890 10/29/18(Mon)16:40:40 No.17722 [R]
not my fault m8
Anonymous 10/30/18(Tue)19:26:41 No.17785 [R]
Why re-invent the wheel when Hydrus is a totally valid option?
It should already do all files and all urls in any post in its supported formats.
Anonymous !cuDH463TvE 10/31/18(Wed)00:53:27 No.17808 [R]
*looks at post timestamp in here and in hydrus thread* Oh, hello CuddleBear! Have tried Hydrus before, not to my taste unfortunately, but good for you if you like it better.
bot named w7 890 11/03/18(Sat)12:01:43 No.18062 [R]
yo OP are u ok no news for 3 fricking dayz
also how dare u upload without telling us (its OK)
link https://www.virustotal.com/#/file/12577f980a326f07fc599f8543109120b2d7eb264cd189fdb8a75cb9f51602ff/detection
the trojan is asleep (the hacker has disconnected) don't wake him up plz
Anonymous !cuDH463TvE 11/04/18(Sun)00:16:45 No.18104 [R]
Yeah, sorry, the latest update is to check if the creator ID is same as last then it'll reuse the last fetched data. This is useful when you have multiple folders of same creator while the contents in them will be different according to pattern in magnificent.txt and mediocre.txt.
There isn't much to update, it's doing the job well. :) Anyone can bring in issues and I'll look into it. But please be competent and use common sense before reporting a problem. I'm not expert and I have no plan to make lot of error handling for something like due to poor connection between your computer and yiff.party. In that instance, just restart CLI to try again. Anyway, I hope everyone can find the tool useful.
Anonymous 11/04/18(Sun)02:20:20 No.18111 [R]
I got this error:
What should I do?
Anonymous !cuDH463TvE 11/04/18(Sun)02:50:37 No.18115 [R]
They aren't error messages (they're state of the settings in yiff.txt). It's finished instantly because it's looking for a number (your favorite creator's ID) in each line in yiff.txt you want to download, but it didn't find any and it has reached to the end of line, so it told you it's finished.
Anonymous !cuDH463TvE 11/04/18(Sun)03:09:12 No.18118 [R]
Updated yiff.bat to yell that there's no creator ID in text file
w7 890 11/04/18(Sun)05:40:32 No.18129 [R]
Anonymous 11/04/18(Sun)06:16:16 No.18138 [R]
Oh, thanks. I put the URL instead of only the ID
w7 890 11/04/18(Sun)12:32:54 No.18157 [R]
>>also how dare u upload without telling us (its OK)
also what was that update
w7 890 11/04/18(Sun)14:46:35 No.18165 [R]
lets make this area a little more private shall we
(incase the payjews are watching)
what's ur email OP
Anonymous 11/05/18(Mon)08:03:24 No.18205 [R]
I got an error with the downloader (forgot to write it, sorry) and then restarted it, but instead of continuing the download it skips the rest of the creator's page and continues with the next one. It would be great if you added the option to check if the files are already downloaded or not.
Anonymous !cuDH463TvE 11/05/18(Mon)09:04:58 No.18214 [R]
Anonymous !cuDH463TvE 11/06/18(Tue)01:28:50 No.18274 [R]
Added a new setting whether to make a "temporary gallery" HTML file or not that will let you view the recently downloaded pictures in your favorite browser.
Anonymous 11/06/18(Tue)01:53:53 No.18275 [R]
i got it!
File "C:\Users\Myuser\Desktop\Patreon\yiff.bat", line 226, in <module>
File "C:\Users\MyUser\Desktop\Patreon\yiff.bat", line 126, in partydownload
FILESIZE = int(requests.get(url + line + "/" + re.sub("#", "%23", line2), st
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\requests\structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
KeyError: 'content-length'</module>
Anonymous !cuDH463TvE 11/06/18(Tue)02:22:35 No.18276 [R]
What was the last url it's trying to download?
Anonymous 11/06/18(Tue)02:41:41 No.18277 [R]
>>18276 It was Jagodibuja's url:
Downloading https://yiff.party/patreon_data/14257600/1371873/yaoyorozu.jpg
Traceback (most recent call last):
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 600, in urlopen
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 343, in _make_request
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 839, in _validate_conn
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connection.py", line 344, in connect
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\util\ssl_.py", line 342, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line
Anonymous 11/06/18(Tue)02:43:05 No.18279 [R]
line 412, in wrap_socket
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 850, in _create
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 1108, in do_handshake
ConnectionResetError: [WinError 10054] An existing connection was forcibly close
d by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\requests\adapters.py", line 449, in send
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 638, in urlopen
Anonymous 11/06/18(Tue)02:44:46 No.18280 [R]
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\util\retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\packages\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 600, in urlopen
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 343, in _make_request
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connectionpool.py", line 839, in _validate_conn
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\connection.py", line 344, in connect
Anonymous 11/06/18(Tue)02:45:25 No.18281 [R]
And finally:
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\urllib3\util\ssl_.py", line 342, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 412, in wrap_socket
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 850, in _create
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", li
ne 1108, in do_handshake
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(1
0054, 'An existing connection was forcibly closed by the remote host', None, 100
54, None))
Anonymous !cuDH463TvE 11/06/18(Tue)03:30:53 No.18285 [R]
Looks like a simple connection problem with yiff.party... keep trying and it'll get through.
Anonymous 11/06/18(Tue)07:35:48 No.18297 [R]
Not anymore, I have restarted the downloader 3 times and it keeps crashing while downloading this file.
Downloading https://yiff.party/patreon_data/9223743/987481/levy2.jpg
Traceback (most recent call last):
File "C:\Users\MyUser\Desktop\Patreon\yiff.bat", line 226, in <module>
File "C:\Users\MyUser\Desktop\Patreon\yiff.bat", line 126, in partydownload
FILESIZE = int(requests.get(url + line + "/" + re.sub("#", "%23", line2), st
File "C:\Users\MyUser\AppData\Local\Programs\Python\Python37-32\lib\site-packag
es\requests\structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
KeyError: 'content-length'
On an unrelated error: https://yiff.party/patreon_data/11216414/1044654/callie1.jpg is corrupted...</module>
Anonymous !cuDH463TvE 11/06/18(Tue)09:27:20 No.18303 [R]
Finally the actual errors I'd want to look into. What was the creator of these?
Anonymous !cuDH463TvE 11/06/18(Tue)09:40:37 No.18306 [R]
Oh, besides, the https://yiff.party/patreon_data/11216414/1044654/callie1.jpg is probably broken on yiff.party too. I've seen several other broken files and these aren't yiff.bat doing. Not much we can do. Meanwhile https://yiff.party/patreon_data/9223743/987481/levy2.jpg looks like there was weird character in filename that yiff.bat hadn't handled properly yet. I wonder what was the filename it originally had.
Also I've recently updatede yiff.bat to handle utf-8 characters I forgot to to include for the new feature (create gallery HTML) I've implemented today. Unrelated to that error btw.
w7 890 11/06/18(Tue)10:16:17 No.18308 [R]
Hey OP whats with the porn pics
also here is the bundle https://openload.co/f/m8PhEV2G4_4/yiffy_client_v1-10.7z
VT link Y10 https://www.virustotal.com/#/file/3a1285190e98903264978d854d41ceb6d72282872ec092140f8e262f34cc3993/detection
VT link Y11 https://www.virustotal.com/#/file/f95dc1861580bd70f8954e86c375b41f13ca965a5f1c39ef6783be3e96a0ed7e/detection
did i miss any version
Anonymous !cuDH463TvE 11/06/18(Tue)11:14:47 No.18309 [R]
Anonymous !cuDH463TvE 11/07/18(Wed)00:26:23 No.18342 [R]
Updated yiff.bat to call 404'd file "Bad link (404): ..." then skip to the next one. You can also ignore this warning by adding them (their filenames) to mediocre.txt too.
Anonymous !cuDH463TvE 11/10/18(Sat)05:10:05 No.18525 [R]
Added proxy (socks) support
Anonymous 11/13/18(Tue)22:13:30 No.18777 [R]
So the program suddenly stopped, I started it up again and after it's done fetching for the Yiff ID, it just stops and says Finished, saying "There are no new posts" but there's still lots of images left (I checked the downloaded files) any help?
Anonymous !cuDH463TvE 11/14/18(Wed)00:50:12 No.18791 [R]
On each launch, it'll start downloading files from the latest on server and as the "Fill =" setting from yiff.txt says to stop at latest file existing on disk, and since the latest post is also already on disk, it finishes with a message telling you there is no new post. "Fill = y" will download the missing files past that.
I'd like to know what have actually happened when it "suddenly stopped" though.
Vera !KhDwBGWuMs 11/15/18(Thu)05:07:22 No.18871 [R]
I mean, I don't know jack about python, but why do you need to use a .bat script?
It seems absolutely useless, and only serves to lockdown which OS can use it. Link me to the most recent version, I'm curious.
Anonymous !cuDH463TvE 11/15/18(Thu)07:50:21 No.18876 [R]
Because I don't use Linux distros and it's in .bat file for convenience. Seriously though, it made things surprisingly convenient. I'm sorry it's for Windows only. It should be possible to extract codes from there and convert codes to not be Windows-specific then it'll become cross-platform. But I'm not gonna give little bit of my effort to make it cross-platform myself. You should feel lucky that I share it here.
The link in OP is always the latest.
Anonymous 11/16/18(Fri)00:51:42 No.18927 [R]
>You should feel lucky that I share it here.
the ONLY reason people want to use scripts like this is because they don't want to install an older version of FireFox and use DownThemAll.
Anonymous !cuDH463TvE 11/16/18(Fri)03:39:09 No.18941 [R]
Improved downloading message "Downloading 0 / ?? MB ..." to show progress using 1MB interval and actually avoid leaving file corrupted on HDD from interrupted download.
Anonymous 11/17/18(Sat)08:06:12 No.19014 [R]
I've yet to figure out how to run this, i have python 3.7 installed, but then i try to run the yiff.bat it opens a console for a second then instantly disappears with no error
Anonymous 11/17/18(Sat)08:25:47 No.19016 [R]
>>19014 figured it out, python wasnt installed in my %userprofile% cuz i installed "For all users"
also it crashed as soon as it started downloading a .psd, would there be a way to exclude that?
and what about files that have no extension, or things like "file.1" https://yiff.party/patreon_data/21718526/file.1
Anonymous !cuDH463TvE 11/17/18(Sat)10:10:27 No.19022 [R]
It shouldn't be crashed by any file format... what was the creator?
Files can be "excluded" by adding their pattern (per line) in mediocre.txt in creator folder, or use a magnificent.txt to only download files by their pattern. If you don't want like .psd then you can add it to mediocre.txt, or if you want only pictures and nothing else you can try with .png, .gif, .jpg, and .jpeg into magnificent.txt.
Anonymous 11/18/18(Sun)07:51:49 No.19070 [R]
>>19022 it was https://yiff.party/910946, as soon as it hit the first psd
Anonymous !cuDH463TvE 11/18/18(Sun)08:18:59 No.19073 [R]
Umm, there doesn't to be a problem downloading the .psd and "file.1". yiff.party tends to have connection problem, so I bet it was connection problem. But I'd probably see the error better if you post the error message either by snapshot or copy/paste from CLI.
I guess I need to rewrite CLI messages for clarification for the first time users.
Anonymous !cuDH463TvE 11/18/18(Sun)09:23:17 No.19075 [R]
Added quick tutorial where to get creator ID
Anonymous 11/19/18(Mon)07:52:12 No.19144 [R]
Do you have any more docs? How can I for example only download certain extensions?
I only want JPGs, PNGs, GIFs, etc. How can I whitelist and blacklist certain extensions? I'd like to blacklist .PSDs for example
Anonymous 11/19/18(Mon)08:25:07 No.19145 [R]
Never mind I'm retarded. I was typing *.psd instead of just .psd
Anonymous !cuDH463TvE 11/19/18(Mon)09:35:38 No.19148 [R]
You got it ;) There's no wildcard support, but the begin/end of line can act like one.
Anonymous 11/19/18(Mon)15:54:45 No.19166 [R]
Any plans to implement a global magnificent.txt and mediocre.txt system? Maybe if in the same folder as the batch file?
For example, I never want to download psds, but if I have several creators I need to copy it into each folder
Anonymous 11/19/18(Mon)16:10:02 No.19167 [R]
Expanding on this, could you implement it so that it also scraped the text from posts?
Many creators store things as links inside their posts. Should be relatively simple.
Anonymous 11/19/18(Mon)17:20:46 No.19175 [R]
I've just spent a few minutes trying to do it myself, maybe to have it write the "title", "body" and "id" parts from the json file to the html file just before the makehtml section, however I'm not familiar with how you're parsing it, so I don't know how to extract that information. I myself would have it save as said before to the html file aswell as to a seperate text file with the "id" as the filename, if the user says yes to saving content in yiff.txt
I'll experiment with it a bit but considering this is your code you might find it easier
Anonymous 11/19/18(Mon)21:43:03 No.19197 [R]
Final post for the night, I'm really novice with python so excuse any issues but I really think you should consider parsing json properly, it'll make it much easier to work with in the future as well as modify.
Here's an example I sort of managed to throw together. I don't understand why you're using a global variable and not just passing info directly to the function, but here's how I modified it.
You pass the numeric creator ID to the download function, this generates a request a parses it, generating the json for every post made by that creator, saved as x. You can iterate through that using a for loop, and then due to it being saved as an object you can then read the post title, body, files, etc.
The only issue I'm having is that the files themselves aren't saved as like a "sub-object". I'd like to iterate through the files but I can't figure out how. If a post has no files it can cause an error for example, or really old posts having no file_names, just an url.
Hopefully you can get some ideas from this but I'd really suggest reworking it to use proper JSON in the future. It'll be so easy. For example, now you have the .file_name you can just check if it exists inside the creator folder, to see if it needs downloading, and you also have the body, which can go into a text file with the post.title as the filename, aswell as go into the html gallery.
Anonymous 11/19/18(Mon)22:00:08 No.19199 [R]
Ok I lied, here's the last post
I wasn't thinking properly on my last post. There's only ever one post file, and only sometimes attachments. This revised code causes no issue.
This gives you the url of the file and the attachment aswell as the filenames, just put the download and file write code inside the for loops and your gold.
Consider using direct json in the future, it's much more readable and easier to work with
Anonymous !cuDH463TvE 11/20/18(Tue)03:07:52 No.19211 [R]
I have previous experience with regex. I'm taking a look at JSON and see how it works, though right now I'm having hard time grasp at it atm.
Yiffy-Get likes to save JSONs on HDD though it's not my thing, however it can be done with little bit of added codes, like below:
jsonwrite = open(foldername + CREATOR_FOLDER + "creator.json", 'wb')
after where "LAST_CREATOR_ID = CREATOR_ID" is at in yiff.bat
I'm not sure what it should look like when saving text posts on HDD. I suppose you can try parse and divide REMOTEFILES.text (r.text in your version) to extract text information and save them as .txt files somewhere.
Updated yiff.bat to read file extension (per line anywhere) in yiff.txt for global file extension whitelist.
Anonymous !cuDH463TvE 11/20/18(Tue)04:05:24 No.19217 [R]
"Shared files =" setting obsoleted by global file extension whitelist
Guy !xZce8u3PEA 11/20/18(Tue)15:26:07 No.19264 [R]
May aswell tripcode for this
I edited an older version of your code, and did a few changes, I'll upload it here incase anyone wants it. I implemented a global magnificent.txt and mediocre.txt, they just have to be in the same folder as yiff.bat
Heres the "readme" of my edit. I changed some things so the line numbers will be different but eh whatever
If you set Gallery HTML = Yes in the yiff.txt file that gets created, it will now rebuild the yiff.html gallery on every run, so you don't have to delete everything or wait for new files. By default, it has the post title, post body, and embedds any attachments. Now that I think about it it will probably error out the ass if said attachment is a .psd or .webm or .swf or whatever but tbh I cba fixing that if the site is gonna close in a few days. Shouldn't be an issue
If you want to stop it including the post body for example, open yiff.bat in a text editor and go to line number 190. I hope it's self explanitory but if not, just remove "\n<br>"+postBody+"" from the partyhtmlwrite.write and you should be good. Scroll up to line 183ish and you can see the data from each post which can be included here
Throw in a magnificent.txt into the creators folder for whitelisting, or have one in the root folder, next to the bat file, type each thing on a new line. These are all CASE INSENSITIVE
Example content of magnificent.txt
This would mean that in this folder, or everywhere, only files/attachments that contain wip, .psd, or diamond in the filename will get downloaded.
Throw in mediocre.txt into the creators folder for blacklisting in that folder, or next to the .bat for global (like typing extensions to globally ignore them) works the same as whitelisting. Both files can work together
Guy !xZce8u3PEA 11/20/18(Tue)15:36:27 No.19269 [R]
Shit, I fucked up on that upload, here's the fixed version. It would previously get an error if a post had no body, or if there was no root magnificent.txt, hopefully this will fix it
Second reply due to line and character limit.
Shit I implemented:
-Proper json parsing for downloads, should be easier to improve on in the future
-This now means that I can also extract post data easier
-Post data is thrown into a html file with the id of the post as the filename and the data in the post as the body. Delete lines 211 and 212 (the with open(blahblah... and the next line, to stop it doing that. Or just delete the files after they download it's not hard I'm sorry don't make me cryplesesto)
-The gallery thing I mentioned above, so it now puts the post data in the gallery too, though with a lot of creators that file is going to be stupidly big. It wouldn't be that hard to make more that one html file, one per folder, with links to each, but meh I can't really be bothered at the moment
-More verbose downloading? I guess? It tells you why it did/didn't download each file from each post
Shit I broke and didn't fix:
-Less verbose downloading, it now no longer shows file progress when downloading. Downloading itself is like 152 onwards so if someone wants to fix that just do it. I tried but I kept getting 403 unauthorized when trying another download method)
-In progress downloads no longer become .part files, they just half-download and end up corrupted if the download fails. (Just delete these files and it will redownload them next run, no need for new posts)
-Probably a lot more
Take a look at my code, see how I did the JSON parsing. I just converted it to a class, and then read the properties
Guy !xZce8u3PEA 11/20/18(Tue)18:43:43 No.19281 [R]
Small edit for quality of life, didn't realize how annoying the html files are
If you still want them downloaded but don't want them to clutter the root folder, replace on line 214(ish):
if not os.path.exists(Path(creatorPath+"posts/")):
This will put them in a new "posts" folder inside each folder.
For anyone doing this you get indentation errors, make sure any space to the left is actually made up of spaces and not tabs.
Or just use this this updated version
Anonymous !cuDH463TvE 11/21/18(Wed)06:36:22 No.19332 [R]
Thank you for sharing. The error popped up saying the "mediocreread" from line 298 is not defined. I'm looking through your code to see what's changed. I can see lot of changes. It might take me little while to understand what's going on. On a side project, I'm starting to understand how can JSON be used. I'm experimenting with it in an independent batch file that work with creator.json I've dumped into certain creator folder. I use it to make HTML file to show text post and load images from disk than online, along with url in text as source. I can see that JSON is better used if trying to extract lot of different types from JSON file.
Guy !xZce8u3PEA 11/21/18(Wed)07:47:02 No.19335 [R]
Hey, sorry about that. I thought I uploaded the fixed version. The problem is it's trying to concatenate the mediocre stuff from each folders file to the "global" mediocre, but if the global mediocre doesn't exist then it can't concatenate each folder to nothing. I've implemented a simple fix for that.
Anonymous !cuDH463TvE 11/21/18(Wed)12:04:38 No.19342 [R]
There are few changes I did to your version http://www.mediafire.com/file/6xvlmqgbxjedy8z/yiffout.bat/file
Not much, but the "partial" file type (.part) is actually important because without it you can expect file to be corrupted once you kill CLI while downloading, or whatever will disrupt its download progress, like internet suddenly went off. I've restored it. Another thing is fix global + per creator mediocreread and magnificentread you can see what my fix looks like!
Anonymous !cuDH463TvE 11/21/18(Wed)12:10:54 No.19344 [R]
My side project is mostly done, http://www.mediafire.com/file/hzj34i5dbvbhsna/makehtml.bat/file
Drop it into creator folder and launch it to build gallery HTML file(s) from JSON (the batch file will download it). Multiple HTML files will be created as continuation using 256 files interval for browser performance.
Here's a funny issue: Any file types will be taken as image, browser will think non-image files are broken images. I don't think I'll bother to do about it yet.
Guy !xZce8u3PEA 11/21/18(Wed)17:01:38 No.19352 [R]
Cool to see how you did it.
One small thing you missed though, on line 209, by removing 2 lines and chaning postInfoFilePath you made it so it saves the html files in the root folder again, however I'd made it so it put them in a dedicated posts folder, otherwise for me it lagged the file browser.
I'll take a look at the gallery generator tho, looks usefull
Guy !xZce8u3PEA 11/21/18(Wed)17:14:04 No.19355 [R]
Uh I think there's a mistake on that release, it's reading and writing the config to the same yiff.bat, breaking it on second run.
I didn't touch it so idk how you're supposed to do it but it looks like you added the word "out" in each filename for some reason
Guy !xZce8u3PEA 11/21/18(Wed)17:27:29 No.19356 [R]
And one final thing, the .path concatenation causes issues.
It's cool, I kind of guessed it would be done like that, renaming after the DL, but I didn't get around to trying it.
The issue is, that filepath is a "WindowsPath", and ".part" is a string, python can't concatenate the two with +.
To fix, convert the filepath to string, and then (idk if this is necesarry but eh I did it) back to path.
Replacing it so it looks like
...pen(Path(str(filepath)+".part"), 'wb') as f:...
os.rename(Path(str(filepath)+".part"), filepath)
Anonymous !cuDH463TvE 11/22/18(Thu)01:05:38 No.19377 [R]
Sorry, it's really dumb code I made when trying to incorporate with original yiff.bat in same directory and load same variables. It should've been just static path like "yiff.txt", "yiff\\", and "yiff.html" respectively. Ah, well!
It's kind of sad that Python will take variable that is only made of numbers as integers and not able to concatenate together with other variables containing str. It adds confusion to the unaware code editors. Happened to me while I was working on my side project.
Anonymous 11/22/18(Thu)01:40:58 No.19378 [R]
Lovely work you're doing. Any update on being able to get the text of a post in .txt? For links and stuff
Anonymous !cuDH463TvE 11/22/18(Thu)02:48:26 No.19381 [R]
Anonymous 11/22/18(Thu)19:54:22 No.19434 [R]
Either drop >>19344 into each creators directory and run it or use my fixed edit https://mega.nz/#!ZsIlxSSZ!Q3U6uCEtbSbfdMCjy1l1fVxfNBw4N5Yn8hI__C3I2VQ which saves all posts as a html file inside a posts folder inside each creators folder.
Anonymous 11/22/18(Thu)21:56:12 No.19443 [R]
Anonymous !cuDH463TvE 11/25/18(Sun)00:00:57 No.19581 [R]
That is odd choice. Unless you mean to download only files by archive file types which can be done with current version when you add their file extension names in yiff.txt. Why does it has to be shared data only?
Anonymous 11/25/18(Sun)04:59:23 No.19590 [R]
Updated yiff.bat to build complete gallery HTML files residing creator folders if "Gallery HTML =" setting is turned on in yiff.txt.
The "yiff.html" at working directory is actually only meant to show the latest pictures from recent download, kind of like daily dose of mine.
Anonymous !cuDH463TvE 11/25/18(Sun)10:22:25 No.19611 [R]
Updated party.bat (>>19344) and yiff.bat with commented out codes for the ability to save per-creator data on disk. Remove the "#" sign before "jsonwrite" instances and they'll work. party.bat will look for and work with JSON file first if it really exists. This allows re-compile HTML files from JSON file regardless if yiff.party is down.
Anonymous 11/25/18(Sun)23:33:10 No.19692 [R]
This new feature allows to download the whole creator.json to be downloaded to the creator's folder, and allows Party.bat to work with it locally instead of contacting again yiffparty's servers, right?
Two little questions; one, is it neccesary to drop party.bat into every single creator's folder, or is there a way to make it run a single time for all creators?
And second; about Fill = and Shared Files in yiff.bat. Using Fill works correctly to have it download only the new stuff by keeping only the last post's files on the folder, but the shared files behaviour is different. It seems to redownload them every time, instead of stopping at the last. Am I doing something wrong, is it an oversight, or is it expected behaviour?
Anonymous 11/26/18(Mon)00:03:10 No.19693 [R]
(same guy than >>19692)
An additional consideration; if an user imports posts of higher tiers that are older than the most recent post; would yiff.party cathegorize them as "newer" even if they have an older date, or as "older", therefore causing yiff.bat to skip them when set to Fill = No?
Anonymous !cuDH463TvE 11/26/18(Mon)00:52:23 No.19698 [R]
Yes, the party.bat will look for local JSON file first, then fetch from yiff.party if it doesn't exists. And yeah it's unfortunate that it's for per creator only for now. I'd probably upgrade it to work across creators to become an offline counterpart of yiff.bat that will only create HTML files from JSON file.
The "Shared files =" setting is obsoleted by file extension whitelist. It was the old way to avoid downloading large files which is typically archived file types. The file extension whitelist can also avoid downloading them as well, so... That setting has since been removed in new yiff.bat, please re-download it from same download link in OP.
The code behind "Fill =" setting that will stop downloading further files is actually expecting a third file in a row to exist then it'll stop. In old yiff.bat, patreon data and shared data were isolated and counter will reset for each. I guess you were experiencing the odd behavior with shared data is due to that, also probably because the certain creator only has one or two shared files.
Again, the latest yiff.bat is always in OP. I occasionally do silent fixes too if I find out the problem.
Anonymous 11/26/18(Mon)01:05:21 No.19699 [R]
Thanks for the reply.
Not sure if you saw >>19693, but additionally, I'm seeing that it's currently not able to get the names from the Yiff.party server, or that gets the creator's name wrong. Is that on Yiff.party's side, or something you might have changed in the bat?
Anonymous !cuDH463TvE 11/26/18(Mon)05:15:18 No.19712 [R]
Sorry, I was busy with other things today. So to respond >>19693 I think yiff.party captures actual ID from Patreon and will download what is available on Patreon and append ID on them. I can see that it can conveniently used to fill the recently deleted post from yiff.party's post flag report and let the yiff.party's scraper do the job. So I assume the higher tier would mean newer as long as the older ones didn't get deleted. I realized that since the new yiff.bat will merge shared data as continuation after patreon data, so yiff.bat will not check for new shared files when it stopped on new post based on "Fill =" setting. Dunno if that's such big deal if "Fill =" setting can always be turned on. It'd also complicate the code to take care of that with "Fill =" turned off as it did for three in a row existence on disk used to overcome the suddenly stopped on "already downloaded" file because of post + attachment duplication which each actually has same post ID + file. There were double "already downloaded" files in extreme cases so it extended to check if a third file exists on disk in a row to stop.
There are two possible scenarios as to why yiff.bat thinks ID is bad (aside from actually bad ID), first trigger is that the ID wasn't found in https://yiff.party/creators2.json, second, yiff.bat actually captured cloudflare's page instead of actual per-creator JSON file and noticed "" in page source (same as those 404 pages, and JSON file doesn't have that) so it tells you the creator ID is bad. You can tell yiff.bat to skip all creators down to that creator to try again when you comment out all the other creators in yiff.txt then restart CLI.
Anonymous !cuDH463TvE 11/26/18(Mon)05:32:12 No.19713 [R]
Oh, yiff.party didn't want to display what was in that "" in my post, it was DOCTYPE html with brackets. Sorry about that.
Anonymous !cuDH463TvE 11/26/18(Mon)06:25:47 No.19716 [R]
What is the creator by the way? If the problem is consistent that is.
Oh, also, you can assign name yourself by the way. It can be any name really. ID is sensitive information, a small typo there can lead to another/non-existent creator. Just be sure ID is correct and then append with non-digit character only, after that, next characters can be digit again and w/e.
Anonymous 11/26/18(Mon)15:20:31 No.19762 [R]
I ended up getting the name myself manually. The ones I saw thing trouble were Rukis (got wrong name, like cslunarin2 or something like that) and h0rs3 (no name at all, just the id numbers).
Posts downloaded correctly for both anyway.
Anonymous !cuDH463TvE 11/28/18(Wed)07:12:35 No.19949 [R]
Added new setting "Log 404 files ="
Updated yiff.bat and party.bat to correctly capture attachment filenames using regex + attachment url for use to build gallery HTMLs to avoid images being displayed as broken, few attachment filenames on yiff.party server are actually broken, so...
Upgraded party.bat to work across creators. Please take it to where yiff.bat is at! Well, technically, it'll look and work with \yiff\ folder. It'll optionally load yiff.bat for global file rejection filters only, and will read sub-directories in \yiff\ as creator list.
Alright, thanks. I'll look into it very soon.
Anonymous !cuDH463TvE 11/28/18(Wed)07:48:55 No.19972 [R]
https://yiff.party/creators2.json actually has some creator names under different ID. What gives? I guess there isn't much to do about it other than have admin look into it.
Anonymous !cuDH463TvE 11/28/18(Wed)09:57:11 No.20021 [R]
I mean... party.bat will look for yiff.txt for global file rejection filter list, not loading yiff.bat for it.
Anonymous 11/28/18(Wed)22:13:06 No.20081 [R]
I want to download "shared only" as some artists like Sakimichan only send their stuff via messages and the only stuff leaked on yiff party is previews...
Anonymous !cuDH463TvE 11/29/18(Thu)05:09:03 No.20099 [R]
Improved file rejection filter system in yiff.bat and party.bat. magnificent.txt will override file type whitelist from yiff.txt, making it possible to accept alternative file types like .zip for certain creator.
It should still be possible to download only those by giving magnificent.txt their file extension names.
Anonymous !cuDH463TvE 11/30/18(Fri)08:14:24 No.20159 [R]
Updated party.bat to extract links to "3114967.raier.txt", etc and will make patrol.txt for the filenames found on disk compared to your personal filter list.
Updated party.bat and yiff.bat to take file extension names from yiff.txt as blacklist if there's exclamation mark ("!") prepended to each. Doesn't have effect until you remove/comment out all other file extensions. This allows downloading other obscure file types and/or broken filenames.
Anonymous !cuDH463TvE 11/30/18(Fri)10:21:31 No.20162 [R]
Yiff.bat no longer has abilities to build complete gallery HTML files, leaving the job to party.bat. Yiff.bat will check if party.bat exists in same directory then save per-creator data "creator.json" on disk ready for party.bat to pick them up. After it's finished, it'll ask you if you want to run party.bat next (press any key to continue) or not (kill CLI to finish).
SeeSaw !EpBHBa/ftQ 12/10/18(Mon)06:03:55 No.20884 [R]
Guess I'll be forced to rewrite my script after all. Still I am amazed you coded this shit in bash, fucking beta :kek:
Anonymous !cuDH463TvE 12/10/18(Mon)09:18:59 No.20891 [R]
Welcome back! And no, it isn't bash, it's batch, Windows thing. Also no one have forced you to rewrite your script. But go ahead, update your script for the neglected OSes. Otherwise, please do tell how can it even recover for the Windows platforms. Let's not talk shit about codes behind mine because it works better than Yiffy-Get frankly. Don't believe me? Try raier, check for duplication and corruption, furthermore, I've made human readable text files for settings'n'stuff and lot of other improvements. If you tried hard enough, your script will probably end up being as good as mine anyway. At least non-Windows users will thank you. <3
7-tard v8.90 12/10/18(Mon)10:02:06 No.20892 [R]
dunno if this works fine on WINE
but my w7 VM is ready
also i apologize for not showing up a few weeks ago
SeeSaw !EpBHBa/ftQ 12/10/18(Mon)10:28:54 No.20903 [R]
I plan to rework the entire downloader to be modular simply put. So it makes adding features more easy, plus the human readability part as well.
Anonymous !cuDH463TvE 12/10/18(Mon)11:09:05 No.20905 [R]
Looking forward to it. I think my script is finished for what I want it to do. But it's always good to have alternatives to fit people need. I've seen people want JSON parser and I've looked into it, considered about it, ultimately not included it. My argument is that yiff.party get information wrong sometime (e.g. filename =/= filename in url in https://yiff.party/787448#p3243120) and the piece of correct information still requires regex to extract AFAIK and secondly it's my personal script, so... ??
SeeSaw !EpBHBa/ftQ 12/13/18(Thu)11:52:03 No.21099 [R]
Yiffy-get the filename handling and stores them in special format that makes it extremely unlikely to get messed up (postid - titlestring - filename as uploaded), it works but it's messy. I want to resolve it somehow but still thinking how.
Anonymous 01/17/19(Thu)04:10:50 No.23340 [R]
What does Log 404 files exactly do? Fiddled with it, couldn't really tell a difference between on and off.
Anonymous !cuDH463TvE 01/17/19(Thu)07:03:03 No.23350 [R]
It's when you weren't able to download the file. If you can download everything OK then it'll not log anything.
Anonymous 01/18/19(Fri)00:59:51 No.23435 [R]
Alright, I've been trying to fix something recently. Trying to download DogBoneArtwork's gallery (2307896), there's a file with a HUGE filename
That will crash yiff.bat immediately, every time. Also, when adding the extension (which in that case es .pe_file_Y3 etc...) to the blacklist makes yiff.bat download everything up to that file (6-7 files) then stop as if there was nothing else under it anymore.
Am I doing something wrong?
Anonymous !cuDH463TvE 01/18/19(Fri)17:05:53 No.23483 [R]
The blacklist/whitelist in yiff.txt is global and works differently than mediocre.txt and magnificent.txt.
Yiff.bat will only take off from yiff.txt as file extensions to avoid matches in the filename before the extension name (like PSD in "December.PSD_01.png"). I haven't checked codes for a while so I'm not sure if yiff.bat cannot handle long or weird extension yet, but you should use mediocre.txt for that instead. Add something like "23194231.https" in mediocre.txt in DogBoneArtwork's folder then restart CLI.
Anonymous 01/19/19(Sat)14:28:28 No.23551 [R]
Hi. WTF is this?
Anonymous 01/19/19(Sat)14:29:00 No.23552 [R]
Hi. WTF is this?
Anonymous 01/23/19(Wed)06:31:08 No.23834 [R]
Say, which version is the latest up to date one? There seems to be a couple different versions linked in the thread.
w7-890 01/23/19(Wed)10:24:16 No.23856 [R]
i also have older versions which are not included here
Anonymous 01/23/19(Wed)23:16:37 No.23914 [R]
Which one works best?
Recently went through the trouble of getting an e621 grabber working on linux and that was a real pain. If you've got an up to date linux version that'd be perfect, though I could probably get a windows one working in wine or just use an old windows XP drive I've got.
Anonymous 01/24/19(Thu)10:47:53 No.23959 [R]
i use KVM and i never actually tried anything Vbox works too
Anonymous 01/25/19(Fri)07:15:25 No.24010 [R]
good to know, you using the one in the OP post then?
Anonymous !cuDH463TvE 01/25/19(Fri)19:48:19 No.24037 [R]
Anonymous 01/27/19(Sun)09:11:17 No.24141 [R]
I'm having trouble using this. Any guide?
Anonymous !cuDH463TvE 01/27/19(Sun)17:15:56 No.24168 [R]
Umm... just run the batch file and read what it says?
Anonymous 01/30/19(Wed)06:40:42 No.24350 [R]
I haven't ever used python before. Do I just put the .txt file in or is there more to this?
Anonymous !cuDH463TvE 01/30/19(Wed)19:44:03 No.24404 [R]
Please change the file extension back to .bat if it's something else. Then you just have to launch it and it'll tell you what to do.
Anonymous 01/31/19(Thu)00:19:43 No.24421 [R]
ty. that was the prob. it was .bat.txt so it was a .txt file
w7-890 01/31/19(Thu)16:00:20 No.24484 [R]
add this plz https://yiff.party/bbs/read/12795#22107
Anonymous 01/31/19(Thu)16:36:50 No.24486 [R]
How do you download rar files? All i'm getting is jpg and png
Anonymous !cuDH463TvE 01/31/19(Thu)19:09:47 No.24501 [R]
Check Yiff.txt again, there are image file types as whitelist. Sorry if it's kind of unexpected after running Yiff.bat several times since its first time.
Anonymous 01/31/19(Thu)22:27:59 No.24519 [R]
I checked the txt file below the bat. The rar was blacklisted. Do I just remove from the blacklist and add to the download?
Edit: so i did that, but I only got some rar files, it didn't download them all. Any suggestion?
Anonymous !cuDH463TvE 02/01/19(Fri)02:10:15 No.24537 [R]
Blacklist isn't in effect until the whitelist has been wiped out. If you blank out both whitelist and blacklist it'll download any.
I guess you have interrupted download at some point since you said you've only downloaded some and feel that there's more, in that case, turn "Fill =" to yes in yiff.txt and it'll download the missing files past the latest files.
Anonymous !cuDH463TvE 02/01/19(Fri)16:16:52 No.24571 [R]
It's in first line of yiff.txt... change from No to Yes
Anonymous 02/01/19(Fri)16:46:07 No.24573 [R]
Sorry lol I didn't notice that. I thought that was a joke line
Anonymous !cuDH463TvE 02/01/19(Fri)17:06:18 No.24574 [R]
Oh, it's alright. I was thinking it'll give away a hint how y/n works.
Anonymous 02/02/19(Sat)02:19:18 No.24611 [R]
is there a way to check embeds for 404s
Anonymous !cuDH463TvE 02/02/19(Sat)03:45:19 No.24616 [R]
"Log 404 files =" to yes in yiff.txt will give you yiff.log file also containing links where 404 files came from. Is that what you mean?
Anonymous 02/02/19(Sat)15:41:50 No.24655 [R]
Any way to download mega download links?
Anonymous !cuDH463TvE 02/02/19(Sat)18:43:17 No.24662 [R]
Party.bat makes a useful file containing every links extracted from posts per creator, you'd just have to open each in browser to download files from outside domains. That's the best my script can do...
w7-890 02/03/19(Sun)06:54:06 No.24700 [R]
dont forget the Megadownloader v1.7
integration is also nice
Anonymous !cuDH463TvE 03/22/19(Fri)21:44:50 No.28860 [R]
Removed setting "Fill =". New way to stop (per creator): get the name of the file you want to stop at, create an empty .txt file with same file name.
Added new feature: Normally yiff.bat will ask for ID for the new creator you want to add, but you can just write that creator name with "lookup." prepended and save (per line) to yiff.txt and yiff.bat will look for ID for this new name. Say, if you know "raier" is on Patreon (and is on yiff.party server) you can write "lookup.raier" to yiff.txt and it'll turn into "3114967.raier".
Anonymous 03/25/19(Mon)23:56:20 No.29095 [R]
What does this do differently to yiff.party then?
Anonymous 04/05/19(Fri)23:55:23 No.30180 [R]
Could use an update. Looks like older links are hosted on dold now.
Anonmon 04/07/19(Sun)16:21:45 No.30317 [R]
Any ideas on what is making this happen???
New ID! I'll check creators list from yiff.party . . .
Traceback (most recent call last):
File "H:\FF Downloads\yiff.bat", line 327, in <module>
File "H:\FF Downloads\yiff.bat", line 137, in remotecreatorlist
creatorlist = json.loads(creatorread)
File "C:\Users\*REDACTED*\AppData\Local\Programs\Python\Python37-32\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\*REDACTED*\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\*REDACTED*\AppData\Local\Programs\Python\Python37-32\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Restart CLI? Press any key to continue . . .
Anonymous !cuDH463TvE 04/08/19(Mon)00:54:34 No.30345 [R]
Updated yiff.bat to download images from new urls
Looks like https://yiff.party/creators2.json just went out of existence. Might it have been moved (anyone have idea where it went)? Temp fix is to write down the creator name after ID. Supposedly you have ID 4979402 which happens to be haychel as its creator name you write like this "4979402.haychel" or "4979402.whatever" and it'll work.
Anonymous !cuDH463TvE 04/08/19(Mon)03:47:35 No.30357 [R]
Updated yiff.bat with a new url where the creator list has gone to.
Anonymous 04/09/19(Tue)22:27:45 No.30597 [R]
one thing i have noticed. is that yiff.bet doesnt grab the high res photos. it grabs some sorta other junk. odd
Anonymous !cuDH463TvE 04/09/19(Tue)22:55:21 No.30603 [R]
Example? Is it different than downloading them yourself?
Anonymous 04/11/19(Thu)22:00:23 No.30818 [R]
New version seems to download ender files that stops scans for other files. Is this intentional? Example 3659577 downloads an ender file after scanning.
Bird 04/11/19(Thu)22:16:00 No.30819 [R]
I love the program! Just needs the ability to opt out of downloading the "_low" copies of files!
Anonymous !cuDH463TvE 04/12/19(Fri)01:00:42 No.30836 [R]
Updated yiff.bat adding more dialogs for new users and not start download suddenly after updating changes to new yiff.txt, giving users chance to check/edit there before downloading with their personalized filters and settings.
You're right, a new version does that. It's a new feature! Ender file will be there (not a downloaded file technically speaking...) after successful download of every file from creator. It'll move once you download more new files from creator. It'll not be there (or not moving) if there's an error downloading a file. It's smarter than "Fill =" setting, willing to try again only on some creators with the error downloading files and the interrupted downloads (usually by killing CLI while downloading).
Now, if you're going to download a lot of things from yiff.party, it should be okay to interrupt and resume like "Fill = Yes" did in previous versions, then conserve storage performance a bit like "Fill = No" did.
Put "_low" in mediocre.txt in creator folder that does that.
Anonymous 04/12/19(Fri)19:54:18 No.30940 [R]
New version is giving the files the ID of the artist in the filename instead of the ID of the file itself, how can I change this? This is making it so files aren't ordered as they were created, like they previously were.
Anonymous 04/12/19(Fri)20:28:42 No.30943 [R]
Disregard, I was running a version from February, thought it updated automatically. Downloaded latest version, it's setting the filename correctly.
Anonymous 04/14/19(Sun)04:25:18 No.31094 [R]
https://yiff.party/668242 LayerIndustries has 30 missing posts
Anonymous 04/15/19(Mon)17:54:47 No.31215 [R]
Couldn't find this in thread, but I'm grabbing stuff from a creator, and along with the actual content, there are a pile of cropped "preview" images. Is there a way to exclude these after deleting them, so the script doesn't redownload?
Anonymous !cuDH463TvE 04/15/19(Mon)20:30:44 No.31228 [R]
If there's a consistent term in file names of these inferior images, then put it in mediocre.txt in creator folder and yiff.bat will not download them again. If they're random in file names then chroming.bat from >>17476 should help before having to constantly copying these file names to mediocre.txt yourself.
Anonymous !cuDH463TvE 04/19/19(Fri)02:03:34 No.31464 [R]
Updated yiff.bat and party.bat to load global rejection filter list along with magnificent.txt if there's "yiff.txt" written in it. Say, you want all files but archive file types from all creators, you'd have
> !.zip
> !.rar
> !.7z
in yiff.txt. But if there's one creator you DO want archive files from, you'd put in
> .zip
> .rar
> .7z
in magnificent.txt in creator folder. But then that will reject literally every other file types for that creator. By adding
> yiff.txt
in magnificent.txt meaning you just wanted more things from that creator. You can still blacklist certain archive files by adding them to mediocre.txt. There are lot of possibilities and a check for "yiff.txt" is bit unusual but it's there for convenience.
party.bat was updated with a hidden feature, edit the batch file and look for "expensive = ". What it does is to scan and log file names of corrupted images to patrol.txt. It will take a while on slower devices.
Anonymous !cuDH463TvE 04/22/19(Mon)10:03:25 No.31694 [R]
Lot of changes were made to yiff.bat and party.bat for small performance boost and covering some other shortcomings. Now handling of missing local files because of older posts that got imported later (by importer with higher tier post access).
Please report back issues.
Anonymous 05/01/19(Wed)19:11:24 No.32311 [R]
superior yet can't download anything besides jpegs
Anonymous !cuDH463TvE 05/01/19(Wed)23:08:05 No.32322 [R]
Right, it doesn't yet, the predefined settings from yiff.txt tells yiff.bat to just download pictures from your favorite creators and nothing else. Yiff.bat is meant to download thousands of pictures and the new ones at your convenience. You should edit the settings in yiff.txt to allow yiff.bat to download other file types.
Yiff.bat is not a scraper.
Anonymous 05/02/19(Thu)13:29:37 No.32356 [R]
edited them, still no result
deleted the enderfiles and all that, still only pictures, nothing else.
Anonymous !cuDH463TvE 05/02/19(Thu)20:31:15 No.32369 [R]
You need to add or delete lines of file extension names in yiff.txt. There's also lines of file extension names after exclamation marks, they're blacklist version that don't have effect until you blank out all the normal file extension names. If you blank out all then it'll download any.
sage 05/04/19(Sat)19:14:10 No.32477 [R]
You can literally do this shit in bash with a oneliner, a few more for interactiveness.
Anonymous !cuDH463TvE 05/05/19(Sun)01:17:32 No.32487 [R]
A crazy long one you mean? Love some example. Can you?
Anonymous 05/12/19(Sun)22:27:26 No.33102 [R]
how do you get it do download by the date?
Anonymous !cuDH463TvE 05/13/19(Mon)00:05:54 No.33108 [R]
It seems to download from newest to oldest as the fetched file list contains post ID in descended order and that descended ID is correlated to older post. I don't think there's a way to download file with its creation date, if that's the other you meant.
Anonymous 05/13/19(Mon)02:04:18 No.33119 [R]
something like that. i downloaded some things, but there are out of order post wise
Anonymous !cuDH463TvE 05/13/19(Mon)06:22:54 No.33128 [R]
It's the way Patreon assigned IDs to their file (technically part of url before the actual filename of a file for now), yiff.party followed suit then yiff.bat in the end. While technically there wasn't ID in filename itself, yiff.bat appended them to filename to work as collision resistance. They can also give you some sense of chronological order. Same ID would mean they're posted at the same time especially from a post of multiple files. There are actually secondary IDs, I think they meant to display file in exact order the creator has uploaded to the post of multiple files. Not sure, and I don't see reason to use it yet, apparently more harm than good, there's sometime a duplication between main file and in one of secondary files, and it's great that yiff.bat happened to not download the duplication because their secondary ID weren't even put into comparison yet. Again, the ID you find in filename coming from yiff.bat is just about collision resistance, then it's up to you to use it as chronological order.
You shouldn't expect ID to order (descended or ascended) exactly as the file's creation date, the file's creation date is created by your OS, and there's no reliable way to fetch original date of a file from the internet, it could be fucked over by a web server, etc, the importer of higher tier would import the older post at later time meaning you'd only get "older" files with newer creation date above other "newer" files.
Anonymous 05/14/19(Tue)05:46:34 No.33177 [R]
Is there a way to get this thing to run in a different location / drive than on the default C drive? My python skills are weak AF.
Thanks in advance!
Anonymous !cuDH463TvE 05/14/19(Tue)08:08:25 No.33181 [R]
Yiff.bat will download files to a folder of same name in current directory (meaning where ever yiff.bat is at and whatever it's named as)
So if you have yiff.bat as Z:\English\Motherfucker.bat then it'll download artpieces to Z:\English\Motherfucker\
Anonymous !cuDH463TvE 06/18/19(Tue)00:59:47 No.35749 [R]
Added a new setting "Show mediocre ="
my shit got baleeted reposting 06/18/19(Tue)10:55:47 No.35786 [R]
Sageman 10/24/18(Wed)13:25:53 No.17328
reason OP's profile pic hey w7 guy plz upload locky
w7 890 10/24/18(Wed)13:31:29 No.17329
>>17326 lick my ass bitch
>>17328 the quick and dirty method no longer works no can do sorry :[ just call the admin
shitbot named w7 v8.9-0 10/24/18(Wed)13:41:06 No.17333
>>17254 >>17291 >>17303 u guys mad
https://www.virustotal.com/#/file/b41ffad900c94872ee6a3d1b099e48a984fceeba1968ba9dd5b4e572579ecc27/detection eat the damn file its safe
>>17329 after that plzkickmyshinymetalazz
WTF.jpg.py has crashed
Orgymania 10/24/18(Wed)13:42:39 No.17334
>>17333 >>17329 streakfagging but i still forgive u
Anonymous !cuDH463TvE 06/25/19(Tue)08:51:57 No.36317 [R]
Updated yiff.bat with less third party dependencies (now there are zero). Uses module named "urllib" in placement of "requests" and murdered proxies support. The "Proxy =" setting in yiff.txt is now useless.
Anonymous 06/29/19(Sat)01:11:29 No.36622 [R]
I updated to the 6/25 version, but when I run it it just keeps saying OFFLINE. I tried having it download a different artist in case it was just a problem with that one, but it still had the same problem.
Anonymous !cuDH463TvE 06/29/19(Sat)03:12:48 No.36630 [R]
It's overkill error message for all error related to trying to fetch creator-related files off from yiff.party (grabbing per-creator data and/or checking name/id for the new creators), I understand that it can't be diagnosed properly. It has to be connection problem generally or invalid connection configuration (the proxy feature was removed in this version). If you edit and remove the "try:" and "except:" from yiff.bat you will see traceback errors that may give you hint what really went wrong, it could just as well be my script is being retarded somehow.
Anonymous 06/29/19(Sat)10:02:17 No.36649 [R]
if "" in REMOTEFILESdec:
NameError: name 'REMOTEFILESdec' is not defined
Anonymous !cuDH463TvE 06/29/19(Sat)12:03:12 No.36651 [R]
You must have removed "REMOTEFILESdec =" line along with "try:" and "except:" by accident. Try again?
Anonymous 06/29/19(Sat)12:26:47 No.36654 [R]
that's because the "REMOTEFILESdec ="" is in the try, so if I get rid of it, it gets rid of that too.
I know it's this chunk that's throwing the error:
REMOTEFILES = partysession('https://yiff.party/' + CREATOR_ID + '.json').read()
REMOTEFILESdec = REMOTEFILES.decode('utf-8')
print("\n OFFLINE")
if I remove the whole thing, I get the "not defined" error I mentioned earlier, and if I comment out just the try and except lines, it complains about an unexpected indent. sorry I'm not super code-savvy, I'm working off a semester each of BASIC, c++, and java from my community college nearly a decade ago, so I don't know a whole lot
Anonymous 06/29/19(Sat)18:32:46 No.36694 [R]
You should give us the ability to specify the python path in the .txt file so whenever you update we don't have to change it again.
Anonymous !cuDH463TvE 06/29/19(Sat)19:35:10 No.36704 [R]
The "try" and "except" is exception handing sort of like "if" and "else" for whenever error comes up. The offline message is coming from "except" because of an error in "try". You need to remove only the "try:" then whole "except:" thing (and fix indentations) so there's no if-else and then you can see the error that exception handling didn't want to show that to you. No worries, you'll get it soon enough.
I can re-up old version if you need it.
Python is one that's reading .txt, so python path in there would be useless. Though if I let batch read there too then it should be possible. What did I change whenever I update?
Anonymous 06/29/19(Sat)20:44:54 No.36715 [R]
Anonymous !cuDH463TvE 06/29/19(Sat)21:44:10 No.36723 [R]
Added a new setting "Python =" then a hidden setting "offlinetraceback =" (inside yiff.bat).
Anonymous 06/30/19(Sun)02:19:25 No.36799 [R]
>>36704 ah ok that makes sense. I also wasn't aware python cared about whitespace, interesting. I tried the 6/25 version and got:
Traceback (most recent call last):
File "D:\BESC\yiff.party\yiff.bat", line 470, in <module>
REMOTEFILES = partysession('https://yiff.party/' + CREATOR_ID + '.json').read()
File "C:\Users\(name)\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 460, in read
s = self._safe_read(self.length)
File "C:\Users\(name)\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 612, in _safe_read
raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(86542 bytes read, 66226 more expected)
and setting the offlinetraceback to true in the new version gave me more or less the same error with a couple different numbers:
Traceback (most recent call last):
File "D:\BESC\yiff.party\yiff.bat", line 481, in <module>
REMOTEFILES = partysession('https://yiff.party/' + CREATOR_ID + '.json').read()
File "C:\Users\(name)\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 460, in read
s = self._safe_read(self.length)
File "C:\Users\(name)\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 612, in _safe_read
raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(86826 bytes read, 65942 more expected)
I tried updating to python 3.7.3, but it still throws the error
Anonymous 06/30/19(Sun)02:45:44 No.36805 [R]
>Please fix path to Python in "Python = " setting in D:\yiff.txt
>Try again!
in my settings: Python = "C:\Python37"
Anonymous !cuDH463TvE 06/30/19(Sun)03:14:59 No.36809 [R]
"Python = C:\" would be C:\Python37\python.exe
Quotation marks not needed even if there's spaces in path. I've uploaded new version imminently, quotation marks was from older version if it wasn't you that added it.
I've seen that error before with proxy and I've killed proxy because of it. Looks like I need work on IncompleteRead issue...
Anonymous 06/30/19(Sun)03:22:48 No.36812 [R]
Sorry, my brain was sort of half-gone when I made that post. Thing is I did try a lot of different combinations (double quotes, single quotes, back quotes, no quotes, folder alone or directly the .exe) and they all lead to the .bat file telling me to fix the path.
Anonymous 06/30/19(Sun)03:57:53 No.36825 [R]
3 hours left
Anonymous !cuDH463TvE 06/30/19(Sun)04:39:08 No.36831 [R]
"Python = C:\" for C:\Python37\python.exe
Anonymous 06/30/19(Sun)08:38:05 No.36851 [R]
Oh fuck, of course, what the hell. I was confident I tried that too. Thanks
Anonymous 06/30/19(Sun)12:19:38 No.36871 [R]
>Looks like I need work on IncompleteRead issue...
does that mean there's nothing I can do on my end in the meantime?
Anonymous 07/02/19(Tue)21:44:43 No.37132 [R]
Haven't read the entire thread so sorry if this has been reported already but this fails to download images that are embedded in the text portion of posts, see: https://yiff.party/patreon/3977850
Anonymous 07/02/19(Tue)22:59:15 No.37145 [R]
Also while I'm at it I think it would be kinda nice if the title of the window/background color of the console changed when it's over, or if it just closed itself. IMO
Anonymous 07/03/19(Wed)18:01:24 No.37211 [R]
First time user, I get
any ideas?
Anonymous !cuDH463TvE 07/03/19(Wed)21:43:25 No.37228 [R]
Guess I'll upload alternative yiff.bat with third party dependencies. No worries, the batch side (not running Python code yet) will fetch these for you. "yiff (requests)" is its name but hardcoded as "yiff" to load yiff.bat's folder and txt.
Try new version (link in OP), the requests module (in old and alternative version) get that error sometime while vanilla get different error, try and see which work better for you. I have no idea as I don't work on urllib module yet. If I do find out why then I'll update yiff.bat. Anyone knowledgeable with this can help.
It'll change color on end of Python (CLI will stay active as batch)
Anonymous !cuDH463TvE 07/03/19(Wed)22:12:14 No.37231 [R]
yiff.bat doesn't look at download links from outside domains. party.bat can help you with that in some way (it compiles line of links detected from post to text file per creator), you'd just have to open each in your browser and download from them. Some may not be just a link to download, and that only you can access to the downloads behind captcha and other weird download methods such as mega.nz that most script can't anyway.
Anonymous 07/03/19(Wed)23:11:46 No.37237 [R]
I think on line 590
if exist "Scripts\pip.exe"
should be
if exist "%Python%Scripts\pip.exe"
Changing it to that made pip upgrade itself but then
[part1] of the pastebin:
after pressing a key
[part2] of the pastebin
Anonymous !cuDH463TvE 07/04/19(Thu)01:45:20 No.37252 [R]
Have you been renaming \Python37\ into something else? Check for the remainders in your code. The first "The system cannot find the path specified." is probably coming from cd, you need to get cd to work.
Updated yiff.bat with "IncompleteRead" fix attempt after I read https://github.com/urllib3/urllib3/issues/1516#issuecomment-469957705
Reintroducing "Proxy =" setting
Anonymous 07/04/19(Thu)02:10:35 No.37256 [R]
oops yeah, it's on C:\tools\python3\, forgot to change the cd command. Downloaded your latest version from OP and still
>Reading yiff.txt . . .
> New ID! I'll check creators list from yiff.party . . .
>Restart CLI? Press any key to continue . . .
Anonymous !cuDH463TvE 07/04/19(Thu)02:36:30 No.37257 [R]
If it happened too instant then there's probably something in my script or your computer, otherwise it has to be yiff.party server.
You can try turn on the hidden setting "offlinetraceback =" to see what's up or try yiff (requests).bat I've uploaded recently.
Anonymous 07/04/19(Thu)02:49:14 No.37258 [R]
>You can try turn on the hidden setting "offlinetraceback =" to see what's up or try yiff (requests).bat I've uploaded recently.
Made offlinetraceback = "true" on line 24 on the bat, result
Anonymous !cuDH463TvE 07/04/19(Thu)03:37:11 No.37263 [R]
Yeah, I don't know who's the real culprit of this, it's coming from urllib module the vanilla and requests module depend on. It seems like the fix attempt has failed... There are slightly different error in requests for some reason, you can try that, or maybe it's all on yiff.party that you gotta wait for it to go away.
Anonymous !cuDH463TvE 07/04/19(Thu)04:32:37 No.37267 [R]
Updated yiff.bat with "(R)etry?" (press R to retry) after every offline error.
Anonymous 07/04/19(Thu)10:00:03 No.37277 [R]
I got a lot of different errors this time after trying the (requests) version
Anonymous 07/05/19(Fri)22:36:59 No.37386 [R]
Look at the posts on that page, the files are all hosted on yiff.party
Anonymous 07/05/19(Fri)22:47:50 No.37387 [R]
Thought of another minor improvement; links to the YP page in the generated html document. Thanks
Anonymous !cuDH463TvE 07/07/19(Sun)01:51:02 No.37484 [R]
Oh, yeah, updated yiff.bat with a fix for patreon_inline pictures. You probably need to delete the ender file and let the new version rebuild a new one. There are two other fixes, since I've dropped requests and use urllib directly, there was problem with unicode urls and the # character, now they'll be url-encode before urllib.
Now there are no more regex codes! They're all replaced with native python codes.
Anonymous 07/07/19(Sun)12:12:56 No.37516 [R]
updated to the 7/7 build and still no dice. I tried a dozen different creators, and one of them worked, but none of the others did, they all threw errors similar to these. still not sure what's going on or why the one creator seemed to work when the rest won't
Anonymous !cuDH463TvE 07/07/19(Sun)14:06:10 No.37524 [R]
urllib is pretty sensitive to bad server or connection. I'll explore way to resume the download so that breaking connection doesn't kill download progress. I learned that not every server support resuming downloads but I hope yiff.party is it.
Anonymous 07/07/19(Sun)16:06:19 No.37529 [R]
I've written my own download-script based on wget, and the last few weeks has been horrible. Basically any download is retried/resumes multiple times before a complete file is downloaded. Dunno if the issue is with cloudflare or yiff.party though...
Anonymous !cuDH463TvE 07/08/19(Mon)08:47:13 No.37570 [R]
Updated yiff.bat with ability to resume download on retry and show 0 / 0 KB progress for fetching data (rather than downloading to disk).
Bad news? The resume download is useless. The resume download code is there in "offline" retry, sure, but for some reason yiff.party server will not let resume download on the creator data, it was exactly where the retry will come in if there's an error. Then somehow yiff.party allows resuming download on the files, but yiff.bat doesn't provide retry on error for them, they'll be skipped.
Anonymous !cuDH463TvE 07/08/19(Mon)09:53:23 No.37574 [R]
Now the resume download has some use. yiff.bat can resume on the interrupted downloads via their ".part" files.
Anonymous 07/10/19(Wed)23:36:33 No.37683 [R]
FileNotFoundError: [Errno 2] No such file or directory: 'yiff\\7874509.cutesexyrobutts\\27418061.https_api-da.wixmp.com__api_download_filedownloadTokeneyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsImV4cCI6MTU1OTc5NDQ5MywiaWF0I.part'
I'm guessing the problem here is the filename is way, way too big. I tried throwing in a file with that name just so yiff.bat can skip over trying to download it but Windows won't even allow me to do that.
This is the file's URL: https://yiff.party/patreon_data/7874509/27418061/https_api-da.wixmp.com__api_download_filedownloadTokeneyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsImV4cCI6MTU1OTc5NDQ5MywiaWF0I
As you can tell there is no extension but it's a PNG image.
Anonymous !cuDH463TvE 07/11/19(Thu)02:58:02 No.37692 [R]
Thanks, updated yiff.bat to truncate long filenames. You'll need to add proper file extension yourself.
New hidden setting "mathhell =" to have fun math screensaver instead of prompting you to press R to retry on offline error. yiff.bat will be in loop until success. The number used in math question is actually number of failed retries.
Anonymous 09/15/19(Sun)20:46:55 No.43493 [R]
Been loving this project, makes things so much easier. Recently though I've noticed that attachments don't get downloaded for at least some artists. I thought it was an issue with Patreon changing things and the importer missing them, but no, if you go to the artist's profile the files are there, just not getting downloaded (https://yiff.party/patreon/106008 is an example of one that doesn't download for me). I've tried to set yiff.bat to download everything I can, ignoring any blacklisting that could be going on, but still nada.
Anyone else run into this? Any ideas?
Anonymous 09/15/19(Sun)21:35:21 No.43499 [R]
To be clearer, it seems to be situations where there are multiple files in the "Media" category (rather than Attachments) that files go missing.
Anonymous !cuDH463TvE 09/16/19(Mon)11:43:04 No.43538 [R]
https://yiff.party/106008.json is what yiff.bat will fetch to see the file list for that creator. If you also look there, there are also missing media files. I wonder if there's another JSON yiff.bat should look into... I'll update once I know what it's supposed to be. Thanks for pointing that out.
Anonymous !cuDH463TvE 09/17/19(Tue)06:26:30 No.43596 [R]
Updated yiff.bat to use HTML files instead. You should delete the ender files then yiff.bat will look for the missing files and download them.
Anonymous 09/17/19(Tue)12:25:57 No.43607 [R]
Thanks for the quick update, seems to be working wonders. Got almost 200 new files with that.
Anonymous !cuDH463TvE 09/19/19(Thu)03:40:52 No.43749 [R]
Updated yiff.bat with new simultaneous download capabilities, the downloads will complete quickly! Lot of code changes so I hope nothing's frell up.
Anonymous 09/19/19(Thu)23:49:02 No.43803 [R]
I found your downloader in early 2019 and came back looking for an up to date (and working) downloader to find you're still around and it's even better. I love you, have some cub.
Anonymous !cuDH463TvE 09/21/19(Sat)08:10:50 No.43920 [R]
I've cleaned up several stupid/useless codes in yiff.bat I made during implementing simultaneous downloads... the last one had to litter the .old files in \yiff\ folder but it will not any more. Clean them up yourself.
w7-890 09/22/19(Sun)06:08:19 No.43982 [R]
looks like i missed alot of versions oh well
also will this work in wine and winXP
Anonymous !cuDH463TvE 09/22/19(Sun)07:15:22 No.43984 [R]
While I specifically said Windows PC with Python 3 install, there's no guarantee it'll work elsewhere but you can try. I'd imagine it'll take just small changes to work natively on other OS too.
Anonymous !cuDH463TvE 09/23/19(Mon)06:41:40 No.44082 [R]
Never mind, updated yiff.bat to work better cross-platform. It being a batch file actually doesn't mean jack to Python (with argument -x) so there's your chance.
Anonymous 09/26/19(Thu)09:01:19 No.44304 [R]
Keep getting stuck with it saying setting up cookies followed by OFFLINE. Any ideas? Updated to latest .bat.
Anonymous !cuDH463TvE 09/26/19(Thu)09:46:20 No.44305 [R]
I noticed that as well. This has to do with cookies... the actual browser also fails to load (I got CloudFlare's "retry for live version" page) after cookies are deleted. yiff.bat will be able to get through after retries. I suppose I could let it save cookies locally so it can be reused later if the issue keeps up.
Anonymous 09/26/19(Thu)11:56:38 No.44308 [R]
You make a good point. Used a web browser I hadn't to view yiff.party, it failed, retried a live version and loaded immediately. Gonna try nuking my cookies now.
Anonymous 09/26/19(Thu)12:03:15 No.44309 [R]
Nuked my cookies in Chrome, still fails with initialising cookies. When I tried to load yiff.party it gave the CloudFlare's retry for live version which immediately succeeded when I clicked retry.
Anonymous !cuDH463TvE 09/26/19(Thu)13:30:33 No.44314 [R]
Yiff.bat will use its own cookies from yiff.party, doesn't grab anything from browser, but you'd see it fail at launch as it doesn't save the cookies to get past of this peculiar issue.
Anonymous !cuDH463TvE 09/27/19(Fri)08:53:13 No.44366 [R]
Updated yiff.bat with a way of saving cookies for next launch. The issue is consistent and annoying.
Anonymous 09/27/19(Fri)20:49:52 No.44393 [R]
Last updated 8 months ago
Breakdown of missing post tiers:
azzai has 154 missing posts
$1.00 - 148 posts $6.00 - 6 posts
Last update 2months ago
tayarinne has 1145 missing posts
$1.00 - 308 posts $3.00 - 449 posts $5.00 - 346 posts $10.00 - 10 posts $15.00 - 32 posts
File is belong Azzai
break link over "lick" draw over this artist
Aerofishtashka could be add?
Anonymous !cuDH463TvE 10/11/19(Fri)21:37:25 No.45763 [R]
Updated yiff.bat for better parallel downloading and removed setting "Log 404 files =". yiff.party has pagination that broke something and admin isn't going to fix the flagging system so the .log file "Log 404 files =" used to make became really useless now. Its removal has helped me clean up codes too. Again I hope nothing's frell up.
i forgot 10/12/19(Sat)17:40:58 No.45802 [R]
the pic
Anonymous !cuDH463TvE 10/13/19(Sun)05:12:17 No.45827 [R]
otter browser user 10/13/19(Sun)10:31:53 No.45839 [R]
just curious who here love otters
Anonymous !cuDH463TvE 10/14/19(Mon)08:19:46 No.45911 [R]
Updated yiff.bat for better CLI color as I found the way to pick the exact color I want. It's funny, Windows 10 wanted to make CLI color easy to read but it has failed to take monochromatic color into consideration. My PC has the custom CLI color for obvious reason but I want to make sure it will look same to everyone.
Anonymous 10/21/19(Mon)01:45:10 No.46460 [R]
Just downloaded the latest version, and now it doesn't find Python anymore. Using the same path as I've always had doesn't work (still works with an older version of the batch), and I haven't had any luck figuring out an alternative that would work.
Anonymous 10/21/19(Mon)01:46:38 No.46461 [R]
Ah, removed yiff.txt and checked to see what it says, it's looking for python 3.8, which I don't think is mentioned anywhere in this discussion. Going to install that and have another go.
Anonymous 10/21/19(Mon)01:49:41 No.46462 [R]
OK yup, that fixed it. So if anyone else has issues running the latest version of the batch file, install Python 3.8.0.
Anonymous !cuDH463TvE 10/21/19(Mon)05:21:00 No.46482 [R]
Yes, you're right, yiff.bat was updated to work with Python 3.8 now. I update the CLI error message a little bit to make it show Python version.
Anonymous !cuDH463TvE 10/21/19(Mon)05:33:30 No.46485 [R]
Renamed party.bat (>>19344) to more.bat and cleaned up cum stain . . . (codes).
> Wanna yiff more?
Anonymous !cuDH463TvE 10/31/19(Thu)23:19:58 No.47478 [R]
Updated yiff.bat with fix for some files with "#" in the filename failed to download.
Here's what happened: urls must be percent-encoded before downloading. The "#" will be turned into "%23" manually as some file names do contain it, then it'll not be mistaken for an url parameter. But during url splitting to encode some string to make them "url friendly" then unsplit, it only happens on string before # and not after it. No wonder some failed to download because the string after # (eventually %23) are not encoded yet.
Vera !KhDwBGWuMs 11/01/19(Fri)05:24:21 No.47505 [R]
imma say something about the script. It more or less works as expected, except in a previous version at least, it would randomly just stop downloading files, without notifying the user why.
Vera !KhDwBGWuMs 11/01/19(Fri)05:29:05 No.47506 [R]
also, would it be possible for the script to parse the posts for URLs for links to MEGA, dropbox, GDrive, etc, and output them to a text file? maybe with the post name?
Vera !KhDwBGWuMs 11/01/19(Fri)05:48:52 No.47511 [R]
Aaaaand it crashes if it hits a colon.
This is a windows problem. The only fix is to have the script intercept this, and rename it the file.
Anonymous !cuDH463TvE 11/01/19(Fri)06:23:26 No.47514 [R]
I think the error will always be same, they'll be errored out sometime for various reasons. At least that the incomplete files will always be resumed, yiff.bat supports resuming interrupted downloads. The broken files on server is up to server owner to fix it. The only progress that is going to reset is when it's fetching new data from creator then interrupted.
I have a separate script for that, it's now called more.bat (>>19344) >>37231 will tell you about it.
Anonymous !cuDH463TvE 11/01/19(Fri)06:35:25 No.47516 [R]
What kind of file system are you on, out of curiosity? Thanks
Anonymous !cuDH463TvE 11/01/19(Fri)06:51:38 No.47521 [R]
Oh, yes, NTFS hates colon characters. Hard to believe that some files will still have illegal characters in, they must be coming from user of different system or renamed post-upload.
Can you tell what creator that has it?
Anonymous !cuDH463TvE 11/01/19(Fri)08:29:17 No.47527 [R]
Sorry, I was sure it was also file system dependent but guess not.
Vera !KhDwBGWuMs 11/01/19(Fri)15:56:03 No.47552 [R]
Michiyoshi. Whenever it reaches a few files it complains. Namely, most of their attached files.
NTFS, by the way.
I'll try move.bat when I get home.
Anonymous !cuDH463TvE 11/02/19(Sat)01:41:56 No.47605 [R]
The colons will be turned into modifier letter colons before saving on disk. Michiyoshi was a fun subject to my experiment ^_^
Vera !KhDwBGWuMs 11/02/19(Sat)01:57:48 No.47608 [R]
I've still noticed it sometimes stops for no real reason. It's incredibly annoying, especially since i have a decent amount of creators in the list. That, along with having to manually reconfirm a retry.
Anonymous !cuDH463TvE 11/02/19(Sat)02:46:19 No.47613 [R]
Oh, you mean that offline and retry thing? It was annoying for sure, yiff.party's connection was terrible from time to time. The traceback message isn't going to say much and it will deadlock the CLI. "OFFLINE" is blanket term for any and all kind of connection problems between your computer and yiff.party and will ask you to retry, it's elegant, and the successfully fetched data per creator will stay until you kill CLI. Keep hitting R should get it through. There is the hidden setting "mathhell =" in batch file that can put it into loop until success, but it seems there was slight memory leak in the code so your computer may not like it running for overnight (that is if the connection keeps failing).
Anonymous 11/02/19(Sat)02:46:33 No.47614 [R]
It's not the connection dropping. I've kinda just made an AHK script to automatically press R every few seconds.
It's just that it'll be running then nothing. No "offline" message, just the last file that got downloaded, and then nothing else.
Anonymous !cuDH463TvE 11/02/19(Sat)03:33:19 No.47621 [R]
Oh, yeah, just need to delete the ender file then it'll look missing files to download again, and if they're already on disk, appears completed but corrupted then it's server's fault to leave them like that.
Vera !KhDwBGWuMs 11/02/19(Sat)04:05:27 No.47626 [R]
I guess.
"del /S *.ender" works well for that
Anonymous !cuDH463TvE 11/02/19(Sat)10:35:50 No.47649 [R]
Updated yiff.bat with return of logging dead links albeit now part of gallery HTML.
Anonymous !cuDH463TvE 11/05/19(Tue)10:34:12 No.48030 [R]
Link to latest version is in OP like usual. I don't have any old versions as I don't save them... no point with saving old ones.
Anonymous !cuDH463TvE 11/12/19(Tue)03:12:53 No.48897 [R]
Added new setting "Patreon cookie ="
This is big update - if you know how to use yiff.party's importer then you can do it! Pledge anyone on patreon then feed yiff.bat your Patreon cookie to download the missing/dead files through Patreon site!
Browser identity (i.e. user agent) and download speed is questionable though. Probably best not use yiff.bat for Patreon if you're paranoid about what will Patreon think of it.
Anonymous !cuDH463TvE 11/14/19(Thu)09:55:23 No.49175 [R]
Added new setting "Yiff.party =" whether should yiff.bat download files from yiff.party or not. That way yiff.bat can turn into Patreon downloader no more.
w7-890 11/19/19(Tue)16:55:45 No.49731 [R]
perfect now we can upload whole patreon stuff in openload (i mean alliance4creativity)
Anonymous 12/02/19(Mon)00:56:49 No.51034 [R]
Can I get instructions in plain english?
Anonymous !cuDH463TvE 12/03/19(Tue)20:39:25 No.51273 [R]
The CLI is short and clear. You just have to run the batch file, it'll tell you what to do next.
Anonymous 12/04/19(Wed)11:13:34 No.51319 [R]
can we have an option to download patreon pages by themselves instead of searching pledges? it would be useful for downloading from patreon pages with lots of free content
Anonymous !cuDH463TvE 12/04/19(Wed)21:10:43 No.51353 [R]
There is a hidden setting that will grab file lists from all favorite creators on Patreon. I added it to grab new files from the creators I stopped pledging.
Anonymous 12/04/19(Wed)22:23:49 No.51360 [R]
can someone explain to a retard like me what this is?
Anonymous 12/05/19(Thu)02:20:08 No.51399 [R]
Alright, I have officially found the limit of this tool.
After seemingly 128 creators added the tool just takes absolutely forever, constantly stopping from working, and generally being a pain in the ass to use.
Anonymous !cuDH463TvE 12/05/19(Thu)07:05:56 No.51415 [R]
I recommend 64-bit Python. Yiff.bat holds file list in memory for all the creators but even then I doubt memory would be skyrocketed, still worth a try. I don't think I have placed such a limit somewhere in there.
For a quick isolation of whatever went wrong (especially for troubleshooting) I usually reduce/replace creator list then restore via undo history of the running text editor. However for a permanent isolation, a separate creator list in different folder & text file will work better. I mean it can be done. yiff.bat can be in different name (e.g. "patreon.bat") and there'll be new text file and folder associate to it, it can have different creator list and settings.
Anonymous !cuDH463TvE 12/05/19(Thu)17:44:01 No.51490 [R]
Hidden settings aren't meant to stand out ;) there are True/False variables inside batch file for silly/debugging purpose, "favoriteispledged =" is the one you're looking for
Vera !KhDwBGWuMs 12/07/19(Sat)22:21:58 No.51723 [R]
Already using Python 3.7.4 64bit.
Will maybe try removing a few or something
Vera !KhDwBGWuMs 12/07/19(Sat)22:23:35 No.51724 [R]
But at 128 entries, it hard stops for at least a few seconds/minutes
Anonymous !cuDH463TvE 12/08/19(Sun)11:22:30 No.51788 [R]
Oh, I think you ran into thread deadlock. Bad code during threading will deadlock the CLI rather than stopping CLI and it would seem to run fine but taking forever to finish, it was waiting for all threads to finish but at least one was killed. There are codes only for Python 3.8+
Anonymous 12/09/19(Mon)00:46:20 No.51841 [R]
can we scan posts for dropbox/mega links and download them?
Anonymous 12/09/19(Mon)18:55:08 No.51902 [R]
Tiffany 12/10/19(Tue)18:58:24 No.51971 [R]
Could someone add an option to download posts in dated folders to separate the content downloaded?
Tiffany 12/12/19(Thu)11:37:34 No.52113 [R]
By the Date Uploaded from Patreon (the date on the posts on Yiff).
Anonymous !cuDH463TvE 01/06/20(Mon)02:00:06 No.55008 [R]
Updated yiff.bat to save timestamps in ender file (only work for files on Patreon site for now), it'll know which file may be edited on server, delete/rename all the associated local file(s) or the ender file to dismiss.
Write "end" in yiff.txt will cut off list of favorite creators for convenient debugging with the above creators.
Yiff.party is very unstable right now ..... OFFLINE (R)etry? Oh, please, (A)lways!
Anonymous 01/08/20(Wed)08:46:47 No.55204 [R]
>delete/rename all the associated local file(s)
how do i pick between the two
Anonymous !cuDH463TvE 01/08/20(Wed)09:24:20 No.55207 [R]
Updated for Yiff.party's new image server domain
It now as well saves yiff.party's timestamps to ender file. You'll get lot of "Edited on server" CLI messages, delete all the ender files dismiss them all. Things has become interesting with timestamp thing when downloading stuff from yiff.party first and then Patreon, there'll be updated post probably with new files or edited files that yiff.party isn't going to have.
Anonymous 01/08/20(Wed)11:11:56 No.55213 [R]
incase some badness happens and the program decides to delete everything, i want to disable that
Anonymous 01/08/20(Wed)12:15:58 No.55215 [R]
Oh, no, yiff.bat can't do either, I mean you must do that if you don't want that message again.
Anonymous 01/08/20(Wed)12:49:50 No.55218 [R]
Yiff.bash when?
Anonymous !cuDH463TvE 01/08/20(Wed)13:30:33 No.55223 [R]
you need Python and argument -x to open it. Windows just have extra benefits from the batch side.
Anonymous !cuDH463TvE 01/09/20(Thu)12:52:42 No.55376 [R]
Example in yiff.txt:
> 9325459.vitrex
> 5238280.tsaiwolf
> Fantia
> 4245.ばっちゃ屋
> Fantia
> 4245.ばっちゃ屋
> Patreon
> 9325459.vitrex
> 5238280.tsaiwolf
Anonymous 01/12/20(Sun)01:13:46 No.55731 [R]
i've been getting this error using more.bat, any idea why?
Anonymous !cuDH463TvE 01/12/20(Sun)03:01:12 No.55737 [R]
I updated more.bat now that it won't encode anything, the unicode characters will display as question (?) marks, exactly what JSONs have now on yiff.party, they used to have unicode escapes that I'd encode, but now they have neither utf-8 or utf-8 escapes but instead displaying ? marks. Not good for non-English speakers. JSON is starting to cripple apart.
Anonymous 01/14/20(Tue)06:27:26 No.56003 [R]
Please add .replace(u"\u2028","") to the list of string methods for the creatorread variables in yp_fantia_filelist and yp_patreon_filelist. This is a unicode character known as the line separator, and it apparently counts for the purposes of the splitlines method. It is used in Japan (and probably other Asian countries) and can sometimes be found in their posts.
Anonymous !cuDH463TvE 01/14/20(Tue)14:27:46 No.56029 [R]
Thanks for sharing! What kind of creator that do that?
Anonymous 01/22/20(Wed)04:29:12 No.56905 [R]
There's some error on the latest version when I comment out all the whitelist
Python 3.8.1
Anonymous !cuDH463TvE 01/22/20(Wed)07:49:28 No.56926 [R]
I recalled that problem and I've shipped a silent fix a while ago for that, are you on the latest version?
Anonymous !cuDH463TvE 01/22/20(Wed)08:22:32 No.56930 [R]
Nvm, I figured out what went wrong, updated.
Andre 01/31/20(Fri)06:13:13 No.58045 [R]
Hi, I discovered another error on the latest version.
Python 3.8.1
Creator that I try to download: https://yiff.party/patreon/10268703
I replace the creator list on yiff.txt with other creator, and it has no problem at all, but this creator make the program error ._.
Sorry if my english is bad, and thank you in advance.
Anonymous !cuDH463TvE 02/01/20(Sat)04:34:30 No.58183 [R]
Updated yiff.bat with HTML builder, it will follow "Gallery HTML =" setting whether to build HTML files or not. It will use assets from yiff.party HTML instead of API, it will rebuild full gallery for Patreon and Fantia creators from yiff.party or via Patreon API (no support for Fantia site yet).
More.bat is obsoleted! Delete JSON files, etc if you don't need them, and delete ender files to force rebuild HTML files, HTML builder relies on ender files to conserve disk performance.
I can see that error in older yiff.bat, the new one doesn't seem to do that, I don't know what was the fix.
Anonymous 02/01/20(Sat)16:45:48 No.58239 [R]
Could you please add a way to have our own folder names for creators? Right now I manually modify the bat with FAVORITE.split('.', 1)[1] in a lot of places just to strip out the id from the folders just so I can find folders faster.
Anonymous !cuDH463TvE 02/01/20(Sat)18:50:51 No.58247 [R]
> FAVORITE = FAVORITE.replace("\\", "/").rsplit("/", 1)[-1]
in yiff.bat
Then edit your .txt and rename creator names, the slash or backslash will now cut off leading characters to be used for folder names
I'll update yiff.bat soon
Anonymous 02/01/20(Sat)19:15:47 No.58249 [R]
Hello, I am getting this error, appreciate any help you can give on it. (P.S. thanks for making this.)
Anonymous !cuDH463TvE 02/01/20(Sat)19:34:04 No.58251 [R]
What was the creator? I'll look into it
jomama 02/01/20(Sat)19:52:19 No.58253 [R]
I have several versions of python installed can i uninstall them and run just the current version?
jomama 02/01/20(Sat)19:57:11 No.58255 [R]
also what does this mean? The ender file will be created/updated after (a witness of) successful downloads of every file from each creator.
Delete the ender file and restart CLI to download the missing (since deleted or previously rejected) local files.
jomama 02/01/20(Sat)21:43:05 No.58263 [R]
yiff.party creator anyway to append a digit to the end of the post_file.png files so it doesn't just keep over writing them
Anonymous !cuDH463TvE 02/01/20(Sat)21:47:06 No.58264 [R]
Thanks! Updated yiff.bat with a fix and one thing (a check for slash/backslash in creator name to use ending as folder name).
Yes, yiff.bat only need the latest version of 3.8.x, you can uninstall all else if you don't have any of the scripts that need older ones.
Ender file helps conserve disk performance. It's like cache, it can be outdated when something is changed (usually by you), deleting it will be like rebuilding a new cache. It's also used to save server's timestamps to help detect updated files on server compared to files on your disk and tell you about it.
jomama 02/01/20(Sat)22:50:54 No.58269 [R]
nvm i see its doing it now i thought it was over writing the post_file files in each folder but it is appending a number to the front of them
choin 02/02/20(Sun)13:25:59 No.58343 [R]
So the long file names thing is still not fixed.
line 443, in echosession
elif not get(onserver, todisk=todisk, referer=referer, threadn=threadn):
line 364, in get
with open(todisk + ".part", 'ab') as FILEwrite:
FileNotFoundError: [Errno 2] No such file or directory:
I've tried fixing this myself but ended up with a file downloaded without extension. But the original URL had it (at least in one case) so it should be possible to properly truncate just the filename.
If you need examples, affected authors are prama and lk00.
Anonymous !cuDH463TvE 02/02/20(Sun)16:11:22 No.58355 [R]
Well, it appears yiff.bat would have problems if it's behind too deep in path name and/or creator's folder having a long name. Yiff.bat allows 200 + 5 (filename + ".part") in filename and Windows allows 246 path characters (including drive letter and slash characters) (TIL about that now), you should figure out where to put yiff.bat in and don't let creators have a long name (they can be easily manipulated in yiff.txt).
choin 02/02/20(Sun)19:13:13 No.58369 [R]
Thanks. If only I knew about it being 200. Maybe at some point it would be better to make it auto-adjustable based on current path, or configurable, or maybe let the user know he has to move the folder closer to the root if some files can't be downloaded.
Some other ideas:
- Long filenames for lk00 could give some idea about how to better truncate such names - e.g. "https*.patreonusercontent.com" pattern would recognize such many of useless names.
- Files without extension could still be named properly automatically since most file types are easy to recognize by their first 4 bytes. This may create issues with recognizing already downloaded files, but it should be possible to just test the existence of any file with specific name and any of supported extensions.
Anonymous !cuDH463TvE 02/03/20(Mon)18:49:07 No.58527 [R]
Yiff.bat will grab new filelist from servers and we'll have to be careful what to manipulate each file in the filelist, renaming them from special variables will not stick for the next time or being same for everyone else.
Anonymous 02/08/20(Sat)06:10:33 No.59053 [R]
The new folder replacement thing you integrated works well, I was just able to replace all the dots in creators names with a forward slash. Thank you!
Art 02/09/20(Sun)21:24:05 No.59250 [R]
Some issue with ender file
29099894.Kupocun - not downloaded, "The ender file will be created/updated after (a witness of) successful downloads of every file from each creator. Delete the ender file and restart CLI to download the missing (since deleted or previously rejected) local files"
No ender file, no folder. How to force download without checking?
Some issue with Python
with clear yiff.txt all working (Some "Added new setting")
With second start - "Please fix path to \Python38\python.exe in "Python =" setting in yiff.txt"
I have a non-english symbols in Python path (User name). Is this make problem? I cleat yiff.txt before each start
Anonymous 02/10/20(Mon)04:44:30 No.59299 [R]
Suggestion: Can download video from yiff.party fantia
Some user have a video, but Yiff.bat not detect it, even though all file format is commented (whitelisted)
For example, this post: https://yiff.party/fantia/posts/256090
Thanks in advance, and sorry if my english is bad
Anonymous 02/11/20(Tue)01:54:43 No.59392 [R]
now every new entry downloads in the same folder
Anonymous !cuDH463TvE 02/12/20(Wed)08:11:43 No.59550 [R]
Just delete ender file will force yiff.bat to check for missing files.
Updated yiff.bat to handle unicode characters now too. It wasn't issue with Python but rather in batch side.
This video's filename ends with .MP4, I guess you forgot to comment out "!.mp4" from blacklist.
If there's no whitelist then blacklist will take place, and if there's not any of it either, yiff.bat will download any.
Fixed! Sorry
Anonymous 02/12/20(Wed)18:58:44 No.59607 [R]
Omg, you're right, my bad. I'm not paying attention to the note that say if all of whitelist is commented, the blacklist will have an effect 😓
Sorry for bothering you.
choin 02/13/20(Thu)11:42:33 No.59686 [R]
Requirement already up-to-date: pip in *\appdata\local\programs\python\python38\lib\site-packages (20.0.2)
Collecting Pillow
Downloading Pillow-7.0.0-cp38-cp38-win_amd64.whl (2.0 MB)
Requirement already satisfied: PySocks in *\appdata\local\programs\python\python38\lib\site-packages (1.7.1)
Press any key to continue . . .
Exception in thread Thread-9:
Traceback (most recent call last):
File "*\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
File "*\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "*\yiff.bat", line 443, in echosession
elif get(onserver, todisk=todisk, referer=referer, threadn=threadn):
File "*\yiff.bat", line 324, in get
echothread(threadn, f"Downloading 0 / 0 MB {echourl}")
File "*\yiff.bat", line 316, in echothread
print(echo, end="\r")
Anonymous !cuDH463TvE 02/13/20(Thu)18:06:42 No.59717 [R]
I'm not sure what could go wrong? I've uploaded a silent fix over silly mistake but haven't seen like this.
choin 02/14/20(Fri)09:40:50 No.59814 [R]
Version I got few weeks ago still works fine, and it doesn't require Pillow.
I went to \Python38\Scripts in cmd and executed "pip.exe install Pillow" - after this I'm not longer getting screen where yiff.bat tries to install Pillow and fail, which was the first part of the issue. Maybe the handling of working path is being done incorrectly.
But the second part stays the same - Exception in thread Thread-9 etc.
Anonymous 02/14/20(Fri)12:07:44 No.59824 [R]
Dogbone continues to break mine in your more recent versions. The filenames seem absurd.
Anonymous 02/14/20(Fri)18:06:01 No.59855 [R]
yea I get the same issue with https://yiff.party/patreon/7874509
theres another traceback i got that said something like "too many values to unpack" and also multiple 404's. looks like its server side. the script worked fine after i commented out the affected creator, but that's not a real solution.
Anonymous !cuDH463TvE 02/14/20(Fri)19:45:02 No.59871 [R]
Added "[:200]" (200 characters limit) to the more filenames I forgot to give, there wasn't a limit without it that couldn't fit in Windows' 246 path limit. Really sorry about that!
I wiped Python installation and reinstalled with the latest Python version (64-bit) from download page and couldn't encounter this kind of error... and besides I think that's unrelated to Pillow by the look of the error. There has to be something else causing that.
Anonymous 02/14/20(Fri)21:29:53 No.59898 [R]
Anonymous !cuDH463TvE 02/15/20(Sat)01:39:27 No.59922 [R]
The CLI color should be blue when running Python (the pure batch side will be yellow on black) but I know that older Windows version couldn't produce it because of no support for the ANSI escape codes where the blue is coming from. I don't have access to the old Windows versions. I think there may be underlying problem in older Windows versions if >>59686 is also on one of these versions too.
Would changing code page to 65001 be causing this? Try put double colons ("::") before "chcp 65001" and then run it.
If no differences, try delete lines of Pillow/PIL related codes and then try again, etc...
For random error with vague message like this I can't ship the fix without being able to reproduce it myself.
Anonymous 02/15/20(Sat)04:38:24 No.59941 [R]
Where can I find the download link to the latest yiff.bat?
Anonymous 02/15/20(Sat)05:08:58 No.59946 [R]
I suppose the OP is getting updated with a recent downloading.
While using it, I have the exact same bug as the yiffy.bat from 6 months ago... it stops downloading without any errors. I haven't dropped my connection at all...
Anonymous !cuDH463TvE 02/15/20(Sat)06:20:52 No.59958 [R]
Yes it's in OP
No idea what went wrong for you though. If only you knew of Python it'll make troubleshooting easier!
Anonymous 02/15/20(Sat)06:41:40 No.59959 [R]
Is it supposed to look like this if I want all of the extensions in whitelist and blacklist to get downloaded?
Anonymous !cuDH463TvE 02/15/20(Sat)06:56:21 No.59961 [R]
Sure, if that's what you want blacklisted ones to become whitelist just did you made them to.
> .whitelist
> !.blacklist
But you can also delete them out of text file if you want to download any rather than just whitelist.
choin 02/15/20(Sat)09:20:12 No.59970 [R]
I'm on Windows 8.1. Guess there is something about CLI not supporting UTF-8 as long as the font used does not support it?
I see from procmon that the exception happens before any UTF-8 file path is accessed though.
Commenting out "chcp 65001" solves the error, but apparently it's not the proper way? E.g. I might end up with broken file names maybe?
Anonymous !cuDH463TvE 02/15/20(Sat)19:07:13 No.60028 [R]
Python can handle utf-8 fine, pure batch can't, chcp 65001 so path to Python can be captured and run it (some users have utf-8 in their names), I think it became unnecessary once variable has got the path to Python.
I've updated yiff.bat that will use code page 65001 to capture path to Python then revert back to code page 437 which was default of cmd.
Anonymous 02/15/20(Sat)20:42:59 No.60037 [R]
offtopic: this is a great tool, do you have plans on making a similar tool for rule34hentai? there's a lot of patreon stuff there too. just a heads up, the admin on rule34hentai is using cloudflare now, which makes it much more difficult to rip (have to add the cookie value every 60 minutes)
choin 02/16/20(Sun)11:44:38 No.60105 [R]
Thanks, it works now.
But found another issue. Either shared files section support is broken or I don't understand how to make it download those files. I think it worked properly like a month ago.
Anonymous 02/16/20(Sun)18:35:42 No.60126 [R]
Wow, I had a slightly older version of this and was having that UTF-8 issue on my Windows 7 box. It made installing missing packages fail and also many downloads. Came here to report it and found the update and it works great now! Thank you for all of your hard work, this is a great script!
As for any improvements, I find that when the script is finished it goes to purple text on a black background. Maybe some color variables that can be edited in the .txt file to customize them would be awesome, setting to output to a log file, and whether to exit when done, wait for pressing any key, or waiting X time before auto running again would be awesome. I plan on running mine on a schedule so a clean exit would rock, but if run in an active console window it would be nice to have it re-run like once a week or something automatically.
Anonymous !cuDH463TvE 02/16/20(Sun)20:36:13 No.60141 [R]
Apparently Yiff.party is on little jittery that trailing spaces could be added randomly in any line in HTML. Yiff.bat went little wonky because of it until I noticed it now... there are few "Already downloaded" stuck messages because of it, delete ender files or waiting for a new file will dismiss it.
Holy fuck, thanks for notifying! Yiff.bat collects assets from HTML than just look through for links, I forgot about shared files. Updated yiff.bat!
Yes, sorry, a little code change can be really bad for somebody else. I was on Windows 10 so I wouldn't know about the others. As for purple on black color when CLI ends, I've wanted to brighten purple a little bit when CLI ends but I can't find a way to do it. I know of ANSI escape codes but can't apply to older text, we are limited to 16 colors. I wonder what's the ideal color for when CLI finishes? I get that purple can be kind of too dark on black.
Why wouldn't you want to edit batch file on your own? You can tweak it with other colors, your ways, timing schedule, etc., and there are hidden features in there as well. I don't think text file should have custom UI settings.
The HTML files ("Gallery HTML =" in yiff.txt) can be used like log file, you get to view the result in browser, or you can leave CLI on to read later.
choin 02/16/20(Sun)23:26:37 No.60166 [R]
It downloads files now. However, I began noticing broken archive (zip and rar) downloads. From both posts and shared files sections. I don't think this happened before.
Anonymous !cuDH463TvE 02/17/20(Mon)00:50:39 No.60172 [R]
Was it download completed and corrupted and it's down to yiff.bat? Error downloaded partial files can be resumed next time, so progress isn't lost. The corrupted file that's not ending with .part in filename is bad news.
Anonymous 02/17/20(Mon)02:48:27 No.60181 [R]
What do I put in whitelist and blacklist if I want to download EVERYTHING including shared files? Thx
Anonymous 02/17/20(Mon)04:29:16 No.60199 [R]
It's trying to download files from dropbox and the filename for those files look like this "29817020.https_www.dropbox.com_temp_thumb_from_token_s_" with a lot of numbers at the end
not sure if it's a bug or intended
Anonymous !cuDH463TvE 02/17/20(Mon)04:57:57 No.60201 [R]
If it's same as what you'd get through normal browser beside an id assigned (for collision resistance) and the 200 filename characters limit it's intended.
Anonymous 02/17/20(Mon)10:27:54 No.60221 [R]
Yes, example is a supposedly downloaded .zip file without .part extension, with size of ~10 mb while it should be ~300 mb.
Anonymous !cuDH463TvE 02/17/20(Mon)10:48:31 No.60225 [R]
Is it down to yiff.bat or? I mean, if it's corrupted on server regardless of what browser or downloader used then there's nothing we can do really.
choin 02/17/20(Mon)11:15:52 No.60227 [R]
No, I could download the same file with browser without issues. And it happened with multiple files during one download session.
I've compared the files and broken ones appear just cut - the download was interrupted by something but the script decided it was downloaded fully.
Anonymous !cuDH463TvE 02/17/20(Mon)11:28:53 No.60228 [R]
Wow, that's so weird
Was it the code page again? I haven't seen problem downloading files but still I don't have older Windows versions to test.
choin 02/17/20(Mon)12:10:37 No.60233 [R]
What do you mean by code page? I'm using the latest version where chcp 65001 is followed by chcp 437. I don't see how this or Windows version are related though. It seems there is no code in the script that would check the download completion and restart it in case it's broken. I mean, it's okay for cases where server can handle all requests without interruptions or freezes, but yiff.party is not responsive in 100% cases for me even in browser.
Anonymous !cuDH463TvE 02/17/20(Mon)13:27:44 No.60242 [R]
Well, if we can be sure it's not code page glitching out the cmd somehow then we'll ignore it. The error message while code page was on 65001 was super strange so I'm not going to rule it out yet for the other strange things from you ;)
There was wonky connection between yiff.party and my pc but haven't seen downloads completed and corrupted where browser could download it fine. I may have missed the certain timeframe when yiff.party tells download is completed prematurely.
As for when it's really completed I think you may be right, it's up to server to decide. Once completed, file will be renamed to drop the .part extension, .part is just way to tell yiff.bat it's incomplete and must be resumed.
If even yiff.bat today still doesn't work very well for you then maybe try disable chcp and then if you want to "resume" the corrupted file you could try put the .part extension back in there.
Anonymous 02/17/20(Mon)19:46:20 No.60293 [R]
If your drive is busy with writing or reading, and/or if you are download something (even if your network isn't being throttled), the .bat will simply freeze and not do anything.
choin 02/18/20(Tue)12:22:41 No.60360 [R]
I've removed both chcp lines from script. Next run I came up with 1 broken file. So I assume this has nothing to do with code page.
From my experience with mass download scripts, if there is no check for download completion, you end up with broken files sooner or later.
Anonymous !cuDH463TvE 02/18/20(Tue)18:22:52 No.60387 [R]
Thanks, so it has nothing to do with chcp
While broken, is it same as downloading with browser? Because even if there is check in place it's impossible to redownload for a better one from same source (corrupted on server).
Out of curiosity, what do these other scripts usually do when they find out download isn't really completed? I may follow their idea.
choin 02/18/20(Tue)19:02:29 No.60393 [R]
I'm checking each case before reporting. If it can't be downloaded properly (e.g. was uploaded with errors on server by original uploader), I'm not mentioning such cases.
The last one I mentioned was 56 mb file while it should have been 123 mb. I removed it and run the script again, and it was downloaded fine this time.
From my understanding, you just copy the remote file size in bytes into variable before the transfer (it seems yiff.bat already does that - it's used in displaying download progress) and compare it with the size of the file after the transfer is complete. If sizes are different - it means something went wrong with the transfer - so you remove the downloaded file and download it from scratch again. I also wouldn't try to resume that since there is no easy way to check whether the connection was cut or if server gave wrong data at some offset.
Also, I never caught the moment it saves broken download during runtime because I don't watch it. Would it be possible to save all the output into the text file?
Anonymous !cuDH463TvE 02/18/20(Tue)21:37:26 No.60412 [R]
Thanks! Sorry to bother you, I was sure maybe you couldn't differentiate between corrupted on server and finished downloading incomplete data but looks like you did, your findings are valuable to me. I'll plan to put a check so whether if fetched data does match to total in the header, those less than total will stay in .part filename for the next time if that's ok with you.
There's module named Pillow that you may notice it installed recently, it loves to complain whenever image file is corrupted, we'll be using it to scan for corrupted files on disk, it's now part of hidden settings, it'll take a while on large collection. However I'm not sure what else can handle corrupted files if they're not images.
Anonymous !cuDH463TvE 02/18/20(Tue)21:57:46 No.60418 [R]
Updated yiff.bat
choin 02/18/20(Tue)22:31:33 No.60424 [R]
I'm not sure why Pillow is even included though. Sure, it's going to be tough for me to check by hand if some image is broken, but I've noticed broken archives because they are easiest to test, then I've noticed broken video files because their size should be much bigger for a specific artist. If you want to make use of it to somehow detect broken files within a huge collections users have accumulated, then it might be a good idea, but otherwise I'm just going to re-download the whole thing from scratch (hopefully later after some additional measures are implemented) and use folder compare or dupe finder software to get rid of broken files.
Other than that, I can't think of any better ways to detect broken download if it's not a file size check. There might be cases where a single byte is altered inside of a downloaded file, but that would be most likely due to hardware issues on user's side. I've been downloading tons of stuff for years using many different tools and I don't remember many cases where a broken file has been downloaded at 100% of its size and a re-download fixed the issue. In absolute most cases the file was broken on server side, so a re-download just confirmed the issue.
I wish there would be tools that could do what Pillow apparently is about - just scan a folder with image files and try to detect files without an proper ending bytes according to its file type, which would mean it was downloaded not at 100%. Never found such a tool in a usable form.
Anonymous 02/19/20(Wed)05:16:21 No.60469 [R]
Oh, well, the reason for having such settings in an external file are so when you update it and I update my copy with your new one, I don't have to go and make all of the changes again. And a log mostly just to show errors. I scrolls past the first stuff so I might miss it. But either way, I am very thankful for your hard work on this! Thank you so much!
choin 02/20/20(Thu)18:28:18 No.60698 [R]
After I've run the whole list from scratch with new version, here are things I can report:
- Downloads could break regardless of the file type. I found broken image and video files on top of archives that were downloaded before chcp commands were introduced.
- There is still an issue where the process just gets stuck or appear frozen, and python.exe produces zero file activity. Maybe this happens when server becomes unresponsive, but that may mean such cases are not handled. E.g. there may be no timeout limitation set to catch such issues.
- It appears that new version just leaves the broken transfer on disk with .part extension so it could be resumed next time. This seems to work fine and I end up with proper files after running it again.
- However there are cases where such .part files are not getting resumed, no matter how many times I run the script again. That happens when .ender file exists for the folder. My understanding is that .ender file should only get created after all files are considered fully downloaded. So maybe there is a bug that makes the opposite possible.
choin 02/20/20(Thu)18:29:52 No.60699 [R]
Nah, I haven't done much work. Thank you for your work and updates.
Anonymous 02/21/20(Fri)00:58:36 No.60727 [R]
Sorry to bother so much, but I seem to be running into a new error. I believe it has to do with some new content as it did not have issues before. I made a pastebin of the error on first run and then the error when it is run again: https://pastebin.com/9aWNGbtK
It does not generate an ender file. The first run claims to have downloaded a file after the error, but has a similar error about "MB" being called before being assigned. The second and subsequent runs have a broken character on the "Exception in thread" line.
As mentioned by choin, I did get through the same user's files in previous versions but had a lot of files named:
So, I deleted them and the ender file so it could try to re-download them and left me with the error.
Thank you for your hard work, it is very much appreciated!
Anonymous !cuDH463TvE 02/21/20(Fri)04:32:34 No.60749 [R]
It tried to download https://yiff.party/patreon_inline/15727305/bd291c3b2731cfa2a1554e36db6e75e230dacdd6.html from https://yiff.party/patreon/3048343#p15727305
Happens again in https://yiff.party/patreon/3048343#p14773846, is also .html
And for some reason we can't see .html's file size from its header "content-length" (who decides on this?) Problem happened because the variable named MB wasn't established this time that the header "content-length" returned none. While yiff.bat will grab HTML as part of reading creator data, the fetch method will do KB without seeing total, only the download to disk method will do MB. There was variable name MB with static value "0 MB" in else statement in older yiff.bat I thought was redundant, now I put it back.
The check for a match to total from the header before renaming to drop .part extension may be best we can have in a while, it should be hard to download anything corrupted that is just you. But for a byte being a little off, for this we can only use tools to verify file integrity, or do what you did, a redownload and compare.
I've added support for scanning archives with a help from 7-Zip. It's now part of hidden settings. Really byte accurate as I've tested. I tried FFmpeg for video integrity but not byte accurate so I dropped it.
Anonymous 02/21/20(Fri)04:43:20 No.60751 [R]
The script will ignore video files with the same filename in the download folder from the same Patreon user.
It's doing that with Nyamota (2383192)
Anonymous !cuDH463TvE 02/21/20(Fri)05:14:42 No.60755 [R]
Amazing filename collision. I'll have to rethink how to save them both. The what seems to be collided in filename was always been just a duplication.
Anonymous !cuDH463TvE 02/21/20(Fri)07:06:58 No.60762 [R]
Oh no, they aren't, check again?
choin 02/21/20(Fri)07:56:04 No.60765 [R]
Huh, I said that I end up with .part files when .ender was created. It turns out they stay like that even after I remove the .ender file. I've checked and found that even though they have .part extension, they were fully downloaded, so they should have dropped the .part extension. There are 2 cases in my folder now and both appear to have the duplicated filenames. E.g. I have a 178kb xxx.Image1.jpg file (note the capital I), and 131kb xxx.image1.jpg.part file. One of them is resized or also has added Patreon logo. So I guess the .part does not get dropped if the file with the same name already exists. In my case both duplicates could have been detected with a case sensitive check, but I'm not sure if there is no similar cases where names are fully identical.
The file size check is enough really. I think file integrity checks are only useful for finding files that are broken on server. But it seems there is still little use in that since I don't expect broken files will be fixed by anyone. I noticed there is a "Flag file" feature but I can't see how it could help anyone. Also, there are many files that can't be downloaded - some give error in yiff.bat because they do not exist (404 error), some get downloaded but they are just xml/json files (under 200 bytes or so) with 403 error texts.
Anonymous 02/21/20(Fri)10:18:16 No.60781 [R]
Still having issues with certain files, to be fair I have a lot of creators. Mind seeing why it wouldn't be happy with these? Especially with Primeleap content that I can view in browser normally.
Anonymous 02/21/20(Fri)10:18:50 No.60782 [R]
Forgot the damn pastebin: https://pastebin.com/raw/PtyHy1U3
Anonymous 02/22/20(Sat)19:57:25 No.60994 [R]
It works for me again! Thank you! That pre-pooukatibg MB with 0 fixed it up! Awesome work!
Anonymous 02/23/20(Sun)17:31:17 No.61082 [R]
pre-filling ... not sure what went wrong with the autocorrect o r whatever ... but I meant pre-filling or pre-setting or pre-populating or something like that.
Anonymous !cuDH463TvE 02/24/20(Mon)11:06:06 No.61153 [R]
Added new hidden setting "collisionisreal ="
Turn it to True and delete Ender file and see if you can get anything that really isn't duplication. The first thing if yiff.bat noticed there's a filename collision (having noticed multiple urls associated to it) it'll compare the size of a file from its header "content-length" to the size of a file already on disk before downloading the differences. I've got so far is duplication in different file sizes though...
Sorry, the description for Ender file is kind of outdated, today Ender will be made after finished downloading of this creator with at least one success download. It'll remember post id to which all the files have been successfully downloaded and not the others with a failed download.
Windows doesn't want to have file with same filename case insensitive to same place and I thought Python would know that. Not sure how it got into downloading part in the yiff.bat.
primeleap_data urls are magically not percent encoded like patreon_data, strange move, but I fix it anyway.
No problem, it was silly mistake
Anonymous !cuDH463TvE 02/24/20(Mon)22:20:46 No.61213 [R]
Small fuck up, I've known since like late 2018 and been like this that there's inferior duplication in certain url for the same filename, recently yiff.bat has downloaded the inferior ones from these urls. If you download new stuff with yiff.bat updated last night you'll probably need delete recent downloads + ender and try again with new yiff.bat I release today.
Anonymous 02/25/20(Tue)06:18:42 No.61245 [R]
Where is the linux version of this scraper?
Anonymous 02/25/20(Tue)16:39:37 No.61283 [R]
Good thing: Now more files got downloaded that were not downloaded before, marked with (2) in names.
Bad thing: The 2 files I had issues with now break the script. I've removed all related files and .ender ones, this is what I get now:
Exception in thread Thread-15:
Traceback (most recent call last):
File "\AppData\Local\Programs\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
File "\AppData\Local\Programs\Python\Python38\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "\yiff.bat", line 483, in echosession
elif duplicate := get(url, todisk=todisk, conflict=conflict, referer=referer, threadn=threadn):
File "\yiff.bat", line 402, in get
os.rename(todisk + ".part", todisk)
FileExistsError: [WinError 183] : 'yiff/387543.janong/3309646.Image1.jpg.part' -> 'yiff/387543.janong/3309646.Image1.jpg'
Anonymous 02/25/20(Tue)16:46:44 No.61284 [R]
It seems both 3309646.Image1.jpg and 3309646.image1.jpg are getting processed at the same time in parallel threads, so the script is unable to detect one is already downloaded by the time the second one gets written.
Anonymous !cuDH463TvE 02/25/20(Tue)22:58:05 No.61315 [R]
Oh! That's it. You gave me hint about race condition of parallel threads. Actually, there's a place that takes same filename but different case as different thing before they are given to parallel threads.
Updated yiff.bat, but the fix is little rough, the conflicting filename will follow original's, like "image" becomes "Image" that the original got there first, if that's ok with you.
Anonymous 02/25/20(Tue)23:23:32 No.61317 [R]
That did it from what I see, thanks. The order is not important to me since it's hardly possible to restore the original order from the data Patreon holds anyway. Each name containing the post id is already enough for sorted presentation.
Btw how should a one-line filter look like if I want to not download all the files with names that contain "badword" (ignoring extension)? I guess I just want a simple example I could follow for mediocre.txt files.
Anonymous !cuDH463TvE 02/26/20(Wed)00:58:32 No.61326 [R]
Yiff.bat will reject all files with filename "containing" any of the patterns (as lines) in mediocre.txt
"badword" from mediocre.txt will match
> 997.superbadword.jpg
> 998.psdbadword.png
> 999.justbadword.psd
Except that pattern with period at the beginning will match the filename "ending with" rather than "containing".
"psd" from mediocre.txt will match
> 998.psdbadword.png
> 999.justbadword.psd
".psd" from mediocre.txt will match
> 999.justbadword.psd
"998.psdbadword.png" got through because didn't have ".psd" in the end.
Anonymous !cuDH463TvE 02/26/20(Wed)02:07:01 No.61330 [R]
And well, if you want to ignore extension then maybe you'll have to look elsewhere to capture, one more trailing character can be effective enough to ignore extension.
Anonymous !cuDH463TvE 02/26/20(Wed)14:36:37 No.61398 [R]
Added new setting "Fanbox cookie = "
For actual Fanbox site, pledge anyone on Fanbox then give yiff.txt your cookie and these creators
This will give you the idea where to put Fanbox creator in yiff.txt
Anonymous 02/26/20(Wed)19:40:50 No.61437 [R]
where do I find the 40 character cookie? also are you sure you mean pixiv fanbox or do you mean fantia?
Anonymous !cuDH463TvE 02/26/20(Wed)20:58:50 No.61444 [R]
Pixiv Fanbox
Well I guess it isn't always 40 characters in length
Anonymous !cuDH463TvE 02/26/20(Wed)21:17:01 No.61446 [R]
Updated yiff.bat that Fanbox cookie isn't always 40 I guess.
Anonymous 02/27/20(Thu)01:20:06 No.61461 [R]
Thank you for making it easier to download.
Does the script handle download URLs from mega and other cloud services too? If not... I think it'd be great if the script could submit a text file with all of those links
Anonymous !cuDH463TvE 02/27/20(Thu)02:09:15 No.61464 [R]
These are arbitrary urls and will likely need your attention, some may not be just a link to download and others may need human intelligence/interaction like there's a download behind captcha.
Yiff.bat can extract them and format them into list, not exactly to text file but, they will be collected and hyperlinked to top section of the HTML for easy to access so you can download them from browser quite easily. Turn on "Gallery HTML =" to enable building offline HTMLs.
Anonymous !cuDH463TvE 03/05/20(Thu)09:47:03 No.62558 [R]
Updated yiff.bat to output \yiff\style.css that you can edit to customize HTML and CLI.
Anonymous !cuDH463TvE 03/06/20(Fri)13:30:00 No.62698 [R]
I've added more color that was stuck in HTMLs rather than CSS.
Things will look wrong with new yiff.bat, need to delete or update style.css yourself to fix. Just for the last time I promise.
I think you're the person that asked about CLI color, you're welcome, I thought .css seems like a nice unusual place to edit to customize CLI too!
Anonymous 03/07/20(Sat)02:15:37 No.62775 [R]
Yes, it is me, the one who requested custom CLI color options. Thank you! I don't know how to do custom names on here, but I am always thankful for your hard work! Thank you again, and keep up the great work!
Anonymous !cuDH463TvE 03/07/20(Sat)04:39:46 No.62788 [R]
"tcolorx" is yiff.bat's main color, it stands for "text color end", it was supposed to be there after text in different color (the "show mediocre =" setting in yiff.txt will show those other color texts), it's now also used as main color of the CLI.
To understand how ANSI escape colors work I'd be following this format "\033[{Y}m", Y can be Background, Foreground, or Reset, multiple combination is possible with semicolon. Insert Y with
> Background: 4X
> Foreground: 3X
> Foreground (bright): 9X
> Reset: 0
X for command's 124 (RGB), 356 (YellowPurpleTurquoise), 07 (BW) otherwise 8;2;0;0;0 for 24-bit color (0 in last 3 segment means black)
Anonymous !cuDH463TvE 03/08/20(Sun)03:58:41 No.62914 [R]
Added new setting "HTML server = "
Updated sample CSS for light/dark theme
HTML will load images whenever is within browser view (lazy load)
Anonymous !cuDH463TvE 03/11/20(Wed)02:18:12 No.63323 [R]
HTML builder includes many new features: avatars+covers, search title, resize images live, toggle sources live, always open links in new tab.
I think HTML got good enough of new features for now.
Anonymous 03/13/20(Fri)02:36:45 No.63574 [R]
Hello! Sorry, I have not had much time to test out the new features. I updated to the most recent version uploaded ( Uploaded: 2020-03-12 19:45:21 ) and am having some issues. I pasted the entirety of the output here: https://pastebin.com/Twtb35en
The script just hangs there, not progressing after that. I gave it a couple minutes in case it was timing out.
Anonymous !cuDH463TvE 03/13/20(Fri)02:56:50 No.63576 [R]
Oh, you're right, I overlook that, the new feature of the recent version is to grab avatars+covers from official sites instead of yiff.party, but apparently they require Patreon cookie (doesn't matter who you pledged), I had Patreon cookie with me.
I'll update to not do that when Patreon cookie isn't available.
Anonymous !cuDH463TvE 03/13/20(Fri)03:08:28 No.63577 [R]
Also there is no timeout, CLI will stuck indefinitely when one thread was killed (got the exception you speak of). That is called thread deadlock.
Anonymous 03/13/20(Fri)07:42:31 No.63602 [R]
heard this thread had cubs lawl
Anonymous 03/14/20(Sat)05:15:17 No.63704 [R]
Thank you for the update! It works great now!
It would probably be nice to have it do a notification or something if it does a thread deadlock ... like if it could detect it and then break out of it or something.
I have not had chance to play with the colors ... I tried a bit and it wasn't responding to my changes. Maybe I was doing it wrong. If I understand correctly, even though there are 5 variables listed under 'cmd', only the 'tcolorx' modifies the colors when the batch file is run in the command line window? Also I think I need a wiki page or how-to article for these color codes. I believe I understand the formatting of the other color variables, but the 'tcolorx' confuses me where it has the number '2' ... Sets background, number 2, reset, foreground (bright), 128 for some reason, set foreground (bright) again ...?
Thank you for the fast fixing of the batch file though! Excellent work! I love how you combined batch scripting & python in one file and I wish I knew python more.
Anonymous !cuDH463TvE 03/14/20(Sat)06:54:17 No.63714 [R]
Thread deadlock after an exception is never a good thing, there are many ways to make it happen and I'll be doing my best to make sure it's not happening.
Alright, if you know of decimal value of RGB channels then you'll get it no problem.
In this example I'll do rgb(16, 0, 32) for background (very dark purple) and rgb(128, 0, 255) for foreground (electric indigo), the tcolorx for this will be "\033[48;2;16;0;32;38;2;128;0;255m" notice where each number go.
9X is just command's set of bright colors, 98 isn't valid, but 38;2;255;255;255 is already white so you can go with 38 for ANSI escape color for foreground.
Also you gotta be on Windows 10 for these ANSI escape color to work.
Anonymous 03/15/20(Sun)00:38:35 No.63853 [R]
D'oh! I run this on a Windows 7 box (less network chatter and weird stuff) that does all of my automated stuff, so that is probably why it wasn't working. But now I understand the codes! Thank you! Hmmm ... it does change the text color when it runs though, yellow for running and purple when done. I really love the changes to the HTML though! Now it is easy to see everything by scrolling at a good size! Keep up the great work!
Anonymous !cuDH463TvE 03/15/20(Sun)16:19:46 No.63975 [R]
Now images in HTML can expand/contract like image boards would. Code stolen from 4chan ...
There was lot of code behind 4chan I've boiled down to codes used to expand/contract images, took long time, but good enough to use for my HTML builder, now I can go to sleep!
Anonymous 03/17/20(Tue)04:00:10 No.64199 [R]
what does "Show mediocre" do
Anonymous 03/17/20(Tue)04:42:37 No.64212 [R]
I'm always getting errors with downloading and I'm pretty sure it has to do with yiff.party temporarily blocking me for excessive requests...
Is there a way to add a delay or something?
Anonymous 03/17/20(Tue)05:32:27 No.64215 [R]
The script finished doing its thing and I've got a lot of files like this
Should I delete it and start over?
Anonymous !cuDH463TvE 03/17/20(Tue)06:06:47 No.64221 [R]
No, they can be resumed next time, that is what .part is about, they're incomplete files waiting to be resumed. Keep them until you find that they can never be resumed for some reason.
I haven't seen yiff.party block users over excess downloading, yiff.party is known to become unstable sometime, maybe that's been what you're experiencing. If yiff.party didn't block you here then it has to be just stability problem.
It will show you files that yiff.bat refused to download because of your filter list.
Anonymous 03/17/20(Tue)08:38:07 No.64238 [R]
I've removed everything in my blacklist and whitelist, is that a wise choice if I want to grab everything or is there a better way to go about it?
Anonymous 03/17/20(Tue)19:01:46 No.64283 [R]
I need to know if I'm using this right.
I run the bat file. I follow the instructions. I wait for the CLI to do the thing and when it says "restart CLI?", I'll close the entire window, delete the ender file and run the bat file again. Is this correct?
Anonymous !cuDH463TvE 03/17/20(Tue)20:18:00 No.64295 [R]
Delete ender files only if you have changed filter list or have deleted something and need to redownload them again. That's pretty much it. Ender file is checkpoint record of previous downloads and for the next time it'll check for new downloads only. The ender file will not remember failed downloads so they can be retried too.
On "Restart CLI?" you can also close it and come back at any time for new downloads.
Anonymous 03/17/20(Tue)21:31:24 No.64308 [R]
Should I be deleting the ender file if there are files with ".part" in the filename?
Anonymous 03/17/20(Tue)22:09:06 No.64312 [R]
Why does this happen? I'm on a 1000/100 line...
Anonymous 03/18/20(Wed)21:27:41 No.64459 [R]
Please add the option to only have 1 active download, I think it'll solve the issue with error downloads
Anonymous !cuDH463TvE 03/18/20(Wed)22:32:37 No.64468 [R]
You can tweak batch file to work with one thread and see if it's working better for you. I still think you'll still likely to get same error plus slower. I'm interested to hear from you how it perform with one thread.
Also has .part file keep growing? That's great, restart CLI multiple times will get everything downloaded eventually.
Anonymous 03/19/20(Thu)04:04:03 No.64518 [R]
Sorry I'm a brainlet, how do I tweak it to work with one thread?
Anonymous !cuDH463TvE 03/19/20(Thu)05:11:34 No.64525 [R]
Find the "for i in range(8):" in batch file, change from 8 to how many threads you like. There are two instances of it, first one is for creator list and another one is for downloading files.
Anonymous 03/20/20(Fri)08:32:33 No.64687 [R]
Hey, I get similar errors as well. Changing the .bat to 1 thread changed nothing to me.
Question out of curiosity: Why does it state "already downloaded" every time? Shouldn't the already downloaded files be added to the ender file? No matter how often I run it, it always says "already downloaded" at the same files.
- - - - 184944.MariArt - - - -
Already downloaded: yiff/184944.MariArt/27230810.ruby_refsheet_by_Mari_HD.jpg
Already downloaded: yiff/184944.MariArt/27230810.ruby_refsheet_by_MariNSFW_HD.jpg
Error downloading: https://data.yiff.party/patreon_inline/27230810/9ea83740adbb87026aa2d3e7ec7c8f70952b89d6.jpg
Already downloaded: yiff/184944.MariArt/22313748.barzillai_commissionWIP5.jpg
Error downloading: https://data.yiff.party/patreon_inline/22313748/c4f8970db25658d06e5b3864e973527f91244a76.jpg
Error downloading: https://data.yiff.party/patreon_inline/22313748/1c3a1dd0f79ed72b3c9f4b7dff7a712f53697031.jpg
Anonymous !cuDH463TvE 03/20/20(Fri)09:13:46 No.64691 [R]
Ender file will ignore the post ID with a failed download, that's why the other downloads associated to that post ID couldn't be remembered.
Anonymous 03/20/20(Fri)20:37:13 No.64758 [R]
some tracebacks ive been getting:
the first one is when i try to use the fanbox downloader with this list:
but "14248629.プニョン" downloads fine when the other id's are commented out
the second one is from the fantia downloader when i use this list:
I have others in the list but the ones above have to be commented out for the others to download
Anonymous !cuDH463TvE 03/20/20(Fri)21:40:49 No.64773 [R]
Thanks! Updated, I haven't pledged any more than one person on Fanbox so the fix for it is untested but looks promising.
Anonymous 03/21/20(Sat)09:09:21 No.64838 [R]
Where do I get fantia and fanbox versions? It seems it doesn't accept any non-patreon id.
Anonymous 03/21/20(Sat)11:02:33 No.64847 [R]
Oh thanks. Somehow I didn't tnink the name part is important, and got more confused when realized the script can become unable to read the .txt if I change its encoding. I supposed only the id is important and the name will be added automatically, same as with patreon. I see now the name can be anything though.
I think fantia gallery html files are not finalized yet? E.g. the part above the logo appears to not work, and I'm seeing "0UTC" where the dates are supposed to be recorded.
Anonymous 03/21/20(Sat)11:49:27 No.64852 [R]
fantia and fanbox downloaders work properly now
but the fanbox downloader doesnt download post attatchments like zips and psds, is that intentional?
Anonymous !cuDH463TvE 03/21/20(Sat)12:54:17 No.64865 [R]
Yes ID and which site they belong to is important part, then you can give them names you like. You can also add slash/backslash in creator name to use ending as folder name.
There are linebreak in Fantia while Fanbox and Patreon don't and I was planning to loop through links split by linebreak, I completely forgot that Fantia has linebreak. The top part in HTML went fuck up where the list of links was supposed to be. Thank you for noticing it, sorry for lack of testing.
Fantia receive 0UTC because they don't have timestamp.
Anonymous !cuDH463TvE 03/21/20(Sat)13:03:19 No.64869 [R]
Yiff.txt probably has .zip and .psd in your filter list (if you haven't removed them yet), try remove them and delete ender file then try again.
However the person I pledged to doesn't have a post with attachments like that so I can't really test myself.
Anonymous 03/23/20(Mon)23:25:51 No.65259 [R]
Getting less error downloads because it's not downloading as many as it can at the same time. I'm fairly convinced that there'll be zero errors with downloading if there's only one instance instead of several
Anonymous 03/27/20(Fri)04:08:14 No.66022 [R]
Nahhh, the more threads, the faster it can go. But the site has times where it loads super poorly due to the heavy load. Heck, the other day, I couldn't even load this page because it would stop part way through. The main website is being hit pretty hard. Sometimes it chokes. The best thing to do is to run the script during low traffic times of day and keep trying until it finishes the downloads. Restricting the number of threads will only mean that you have to wait for each and every file to time out and die before moving on to the next, where you could be timing out on 8 or so at once and get your results fast & be prepared for a new run. I am a fan of multiple threads. Use that bandwidth when you can!
Just my two cents.
Anonymous !cuDH463TvE 03/30/20(Mon)05:19:43 No.66575 [R]
Updated yiff.bat to check for creators you still have access to their paid content on Patreon for a month after you stopped pledging. Not sure how reliable is it though.
Well said! Yes, in my experience, less threads to have less errors is just placebo.
w7-890 03/30/20(Mon)14:43:38 No.66649 [R]
hey umm sorry for ruining the party but how do i download thots on YP using the script (to be clear this is NOT a request thread)
Anonymous 03/31/20(Tue)02:02:06 No.66749 [R]
There are less errors with downloading with one thread and there'll be zero errors if we can only have one instance of downloading
Anonymous 03/31/20(Tue)02:54:17 No.66761 [R]
I do seem to have a pretty epic network connection and have things run at a time that is typically "at night" for most EU locations and probably US too, so it is likely a great time to do things. I have great success with downloading using all of the threads and seem to never end up with .part files either.
My best thoughts would be that maybe an option in the yiff.txt that allows one to set the maximum number of threads could satisfy the varied types of connections.
Thank you! I was worried that the tone would be an undesired one, but I mean well. I imagine that if some have slower connections, that one thread might grab a download and the others would time out and give more errors. Not everyone has a decent connection, I suppose. Sometimes slow and steady wins the race?
I do love my multiple threads though.
Are you asking how to run the script to download your favorite items from yiff.party, or are you asking how to find good stuff on YP? I found this site that has "top" people in different categories ... https://graphtreon.com/top-patreon-creators ... maybe that helps? Otherwise, I think this is more of a "if you know what you want, this helps you get it" - and in an automated fashion so you don't forget or miss stuff.
Anonymous 04/01/20(Wed)01:38:46 No.66965 [R]
It has nothing to do with how fast or stable your connection is. This site will temporarily block you if you make too many excessive page loads.
Anti-Yiff 04/02/20(Thu)15:02:20 No.67355 [R]
Wtf?!?! You just insulted KDE fans, fucking scum?!?!
tardorito !elUzZM2K/o 04/02/20(Thu)18:17:23 No.67389 [R]
Anti-Yiff 04/02/20(Thu)08:02:20 No.67355[R]
Wtf?!?! You just insulted KDE fans, fucking scum?!?!
mirroring incase comment is gone
hey atleast he didnt insult winblows fans lololol
(i cringed while making this posts my hands shake the keyboard)
i was expecting some chaggotry (kek) but thanks for (sort of) giving me useful info
also does using a hyper V affect download speed
Anonymous 04/04/20(Sat)03:42:30 No.67682 [R]
I doubt that using Hyper-V would change much, but it all depends on your network configuration. I have seen a lot of crazy stuff. It shouldn't normally, but everyone has a unique setup.
As far as the site blocking you, if you run into the issues where you can't download, then you should try loading the site or one of the images manually and see if it gives you an error. In my experience, whenever I have seen issues, the site also was timing out when loading. But not consistently. I would open, say, 5 to 7 images in tabs and less than half of them would just time out after a long time when the others load slowly. This is not a block, this is the site's web traffic being saturated and not being able to respond to all of the requests.
But if you see a legitimate block (will look something like this: https://us.v-cdn.net/6031209/uploads/drupal_attachment/files/2019-06/26/99/2699b06143a761568086e2a4278e290d.png - or you will be able to load the site though your cellular network, but not your computer) - then that shows that they implemented a blocking situation. It would be good to know if they did. They are behind CloudFlare, so you would probably get a CloudFlare notification if they were blocking you.
Anonymous 04/08/20(Wed)00:11:35 No.68493 [R]
YP won't permanently block you from accessing the site. It'll temporary block you for a while if you are making too many excessive pageloads. That's why everyone is getting 'error downloading'.
Anonymous 04/09/20(Thu)23:24:28 No.69018 [R]
File sharing doesn't work!!!!!!!!!!!!
Anonymous 04/13/20(Mon)10:06:59 No.69822 [R]
Is there a way for it to search folders in the creator folder? I can just sort out different type of images or put images that I don't want in folders then it will auto skip download these images.
Anonymous 04/13/20(Mon)14:18:07 No.69844 [R]
I really think error handling should be improved. Right now there are these problems:
- Some files fail to download due to timeout. Must remove ender file and restart to try again, and they may still fail. Eventually they will download though.
- Error messages for 404'd files and timeout cases are same: "Error downloading".
- A download may get stuck for hours.
Things that I think may fix these problems:
- Differentiate between failed download cases. Detect if there is an error or if there is a timeout.
- If it's a timeout - try again repeatedly until some response is received.
- If it's a permanent error - create some empty file and mark these files as absent.
- During future runs, don't attempt to request these files if they are marked as absent. This will save the time because each request can take long to process.
- Report the error code properly in the log.
- Try to detect timeouts for running downloads and restart/resume them when they get stuck.
Anonymous 04/13/20(Mon)16:15:51 No.69860 [R]
Have one instance of downloading only. Solves 99% of the problems.
Anonymous 04/13/20(Mon)17:36:04 No.69875 [R]
Correction, downloads should be restarted upon not just detected timeouts but any non-404 errors I guess. Right now I'm getting a lot of 502 in browser.
I imagine sending several requests at a time during high load would indeed decrease reliability of each individual response. But it won't solve all the problems. Error detection should still be there.
Anonymous 04/13/20(Mon)18:55:13 No.69885 [R]
Anonymous !cuDH463TvE 04/13/20(Mon)22:47:41 No.69962 [R]
I'll explore a way to retry error downloads without need to restart CLI. In the meantime you can restart CLI several times to be sure you have downloaded everything. You don't really have to delete the ender files after an error download. Ender will not save them.
Mediocre.txt can prevent a file from downloading next time, be it truly mediocre to you, or a server-side corrupt/unavailable file you want to skip.
The error downloads will be logged to HTML if you don't mind the HTML file to work like log file. "Gallery HTML =" will enable building offline HTMLs.
Regarding download getting stuck for long time you can try cut internet and it'll timeout instantly. This will be useful to prompt a offline message right away, not so much when downloading files though, until I implemented a retry for them anyway.
Anonymous !cuDH463TvE 04/14/20(Tue)04:47:44 No.70023 [R]
Updated yiff.bat listening to keypress R and S while CLI is running. Requires system with ANSI escape support like Windows 10, MacOS, idk about Linux.
> Press R to always request retry or always skip error downloading files.
> Press S to skip once.
Anonymous 04/15/20(Wed)00:34:51 No.70216 [R]
Any chance for a feature that rips zips, too?
Anonymous 04/15/20(Wed)01:48:01 No.70235 [R]
I didn't blacklist anything. Am I retarded and the bat is auto-unzipping? Otherwise, I'm not to sure how to check the blacklist (if there was one made).
Anonymous 04/15/20(Wed)02:04:35 No.70240 [R]
Gotcha! Figured it out, thanks. Kudos on the great app; jdownloader started to suck shit.
Anonymous 04/18/20(Sat)12:23:33 No.71038 [R]
Trying to figure this out and I'm lost. Windows 8.1 here. First, it doesn't seem to react at all when patreon is processed. Second, it reacts when trying fantia, with changing colors and window title. But fantia doesn't seem to work now ("OFFLINE") even with older version, so maybe it got broken.
Anonymous 04/18/20(Sat)18:52:06 No.71095 [R]
How do you restart the CLI?
Anonymous 04/23/20(Thu)09:17:07 No.72098 [R]
I was wrong, there are no issues with fantia, just bad configuration in txt.
Still, from what I see the R/A keys work only when there is a prompt for it - e.g. when script reports OFFLINE thing. I don't see any change if I just press the keys during a normal run, so I don't know if those download errors are treated differently from before.
Anonymous !cuDH463TvE 04/23/20(Thu)10:14:20 No.72104 [R]
It's for the requested data that are required before script can continue.
Anonymous 04/27/20(Mon)16:34:18 No.72807 [R]
Please give us the choice to download one thing at a time only
Anonymous !cuDH463TvE 04/29/20(Wed)05:17:22 No.73104 [R]
Just reminder that yiff.party CAN be unstable sometime and that there's nothing wrong with yiff.bat. Please keep trying or come back later. There are no performance difference between browser and script for as far as I know.
Anonymous 05/12/20(Tue)14:53:04 No.75746 [R]
Anonymous 05/15/20(Fri)02:09:41 No.76494 [R]
Here's updated proxy code to support SOCKS5 proxies with authentication (and trailing backslashes no longer break the port number):
if "socks5://" in proxy and proxy[10:]:
~ ~ if not ":" in proxy[10:]:
~ ~ ~ ~ print(" PROXY: Invalid socks5:// address, it must be socks5://X.X.X.X:port OR socks5://user:firstname.lastname@example.com:port\n\n TRY AGAIN!")
~ ~ ~ ~ sys.exit()
~ ~ if not "@" in proxy[10:]:
~ ~ ~ ~ address, port = proxy.replace("socks5:","").replace("/","").split(":")
~ ~ ~ ~ socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, address, int(port))
~ ~ else:
~ ~ ~ ~ usr, pw, address, port = proxy.replace("socks5:","").replace("/","").replace("@",":").split(":")
~ ~ ~ ~ socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, address, int(port), username=usr, password=pw)
~ ~ socket.socket = socks.socksocket
~ ~ # The following line prevents DNS leaks. (( src: https://stackoverflow.com/questions/13184205/dns-over-proxy ))
~ ~ socket.getaddrinfo = lambda *args: [(socket.AF_INET, socket.SOCK_STREAM, 6, '', (args[0], args[1]))]
Swap out the squiggles for spaces.
Anonymous 05/15/20(Fri)16:03:21 No.76898 [R]
if exist "Scripts\pip.exe" (echo.) else (color %stopcolor% && echo PIP.exe doesn't seem to exist . . . Please install Python properly^^! I must exit^^! BYE pause>nul && exit)
This line is missing a conditional, making the script terminate instantly. It should be:
if exist "Scripts\pip.exe" (echo.) else (color %stopcolor% && echo PIP.exe doesn't seem to exist . . . Please install Python properly^^! I must exit^^! BYE && pause>nul && exit)
Anonymous 05/15/20(Fri)17:40:37 No.76955 [R]
This has been a nuisance for a while... there is something about the "choice" function call in the retry function that makes my scraper hang every time after it's finished downloading everything. Whenever I update yiff.bat I just delete this entire code chunk:
~ ~ ~ ~ else:
~ ~ ~ ~ ~ ~ title("OFFLINE", True)
~ ~ ~ ~ ~ ~ print("OFFLINE (R)etry? (A)lways")
~ ~ ~ ~ ~ ~ if choice("ar", True) == 1:
~ ~ ~ ~ ~ ~ ~ ~ offlinepromptx[0] = True
AND I have to disable ansi_esc, i.e: ansi_esc = False
After that, it works fine... I'm really not sure what the problem is.
Anonymous 05/15/20(Fri)23:33:57 No.77132 [R]
That won't change anything. It's still hammering yiff.party's server and trying to grab as much as it can. We need a proper 1 by 1 download.
Anonymous 05/16/20(Sat)01:11:58 No.77160 [R]
it works for me tho. I change 8 to 1 and it download 1 at a time. There's 2 same code, maybe you edit the 1st one. Just search from the bottom
Anonymous !cuDH463TvE 05/16/20(Sat)04:30:54 No.77213 [R]
Thanks again. Updated yiff.bat with small fix. Also does the weird issue with choice function call have to do with offline message you're getting? There's a focus conflict between two input prompts and you'll need hit a key twice so the right one can receive a key it was expecting. For example on Windows 10/MacOS accept A, R, S, and X keys (ansi_esc = True) and older Windows versions accept only A and R keys (ansi_esc = False), one prompt does A, R and then there are S, X in separate prompt. Sometime when you hit R after there's an offline message and it might not work because the S/X prompt was in focus and was expecting S or X but R wasn't in it, somehow it loses the focus and A/R prompt takes turn. Yes I know it's unreliable but seems to work. Anything that use os.system() on top of another can be problematic, especially os.system("choice ...") THEN os.system("title ..."), ansi_esc was born for that.
Now you can see there is
> print("\033]0;" + echo + "\007\033[A")
when ansi_esc is True and
> os.system("title " + echo)
when ansi_esc is False
And Windows versions less than 10 don't support ANSI escape code. os.system("title ...") completely breaks os.system("choice ...") but os.system("choice ...") on top of another os.system("choice ...") seem to be just focus problem that somehow they can take turn when one key wasn't meeting prompt's expectation, not big deal.
Anonymous 05/18/20(Mon)04:00:47 No.77925 [R]
Yes, there definitely does seem to be an input prompt conflict. Apparently if you activate auto-retry mode with X and ALSO hit R to retry whenever a failure occurs, you can get it to work somwhat reliably. With ansi_esc off completely (and without code modifications) it appears to need to download all the files it needs (before scraping creators) 3 times in a row, which I do not understand at all.
Anonymous 05/18/20(Mon)04:07:14 No.77930 [R]
Thanks for this. I updated to the new one but now it's not working. I'm trying to run it off a network shared folder as a mapped drive. Did something change in the bat where it checks where it's running from? With realpath it's trying to \\sharedfolder\\folder\\ with os.chdir(batchdir) error line 30.
I edited the batchdir = (os.path.dirname(os.path.realpath(__file__)) + "\\").replace("\\\\", "\\") to abspath and it sees it as mapped drive X: but then something else errors out on line 2085 on cookiejar.py.
Anonymous !cuDH463TvE 05/18/20(Mon)09:21:26 No.78026 [R]
> batchdir = os.path.dirname(os.path.realpath(__file__))
was changed to
> batchdir = (os.path.dirname(os.path.realpath(__file__)) + "\\").replace("\\\\", "\\")
I added a trailing backslash to treat it as folder, but I guess this is a poor implementation. I discovered it was down to batch side of the code, so I reverted the code for variable "batchdir" in python side and applied to batch side. It's not going to make a difference if you run batch file. Will your network storage be okay with forward slash or does it can't really have any of the trailing slashes?
Must be pagination, it's normal. It will happen to when you're scraping from Patreon, Fanbox and yiff.party's Fantia where they are split in pages.
Anonymous !cuDH463TvE 05/18/20(Mon)10:04:20 No.78043 [R]
Also A, R and S, X do different things.
S or X will not make that offline message go away, A or R will. The offline message is for the required data yiff.bat can't continue without them, while S and X is for the optional downloads so they can be retried rather than restarting CLI and do all over again just for those. I made S and X available all the time to be concurrent with other task and don't have to hang on us for an input, especially for optional downloads, I wanted a quick toggle for little things.
The S/X input prompt had a R instead of X (R/S input prompt), but that's another os.system() problem, every letters prompted had to be separated so R in R/S turned into a X.
Anonymous !cuDH463TvE 05/18/20(Mon)10:36:41 No.78048 [R]
Anonymous 05/18/20(Mon)18:09:21 No.78211 [R]
Sadly still not working. Guessing the line 2085 error is because it's not concocting the path and file name together properly?
Traceback (Most recent call last): File "Z:\YIFF\yiff.bat", line 1261, in <module> cookie.save()
File "C\Programs\Python\Python38\lib\http\cookiejar.py", line 2085, in save with open(filename, "w") as f: PermissionError: [Errno 13] Permission denied: 'yiff/cookies.txt'
going backwards through the code I'm assuming it's on line 1198 cookie = cookiejar.MozillaCookieJar(mainfolder + "cookies.txt") referencing mainfolder
mainfolder which is on line 29. mainfolder = batchname + "/" and
batchname which is on line 25 batchname = os.path.splitext(batchfile)[0]
Been fiddling with the main variables with little luck. I should read up on python lol
Anonymous 05/18/20(Mon)19:57:34 No.78259 [R]
im kind of a noob in python,what's the command for running a batch file?
Anonymous !cuDH463TvE 05/18/20(Mon)21:09:51 No.78289 [R]
Anonymous 05/18/20(Mon)23:49:52 No.78370 [R]
Thanks for the assist. I reset my entire network and all the security settings on the folder and it's up and running again.
Anonymous 05/19/20(Tue)19:27:30 No.78732 [R]
Is there any way I could have it retry failed downloads (i.e. downloads that don't 404 but just weren't able to finish before a server error)?
Also, is there a way I can limit how many concurrent downloads are going on at once?
Anonymous 05/21/20(Thu)03:13:11 No.79454 [R]
I was sifting through the downloaded stuff and noticed that when there's an iframe embedded link like a youtube It doesn't pull that data to the gallery other than that everything works lovely I think the only time it's been bratty is when I accidentally 2 of the same id# a few times on the list.
I thought it was automatic. I recall seeing it saying that if it failed it will retry.
Anonymous 05/21/20(Thu)12:20:40 No.79595 [R]
Whenever I add a new line (12345.name), my usual course of actions is:
a. Run the bat.
b. Check for .part files. Of course there are (still) there.
c. Remove .ender file.
d. Go to a.
The cycle repeats itself until there are no more .part files. Around 10 times lately.
Maybe if .ender file would only be created if there are no .part files (no download errors), it'd be somewhat easier to manage.
Anonymous 05/21/20(Thu)15:59:15 No.79666 [R]
Anonymous 05/21/20(Thu)18:29:49 No.79717 [R]
Anonymous 05/21/20(Thu)21:02:57 No.79773 [R]
i stand corrected
on the other hand, this tool could really use some cohesive documentation
Anonymous 05/22/20(Fri)11:09:38 No.80022 [R]
From my experience, failed downloads (.part) remain untouched if I don't remove the .ender file - the log just says it's tripped and there are no info about trying to download other files. I thought maybe because their URLs are not in the beginning of the page, so the .ender is getting tripped earlier than some older unfinished files could be checked.
Anonymous 05/22/20(Fri)14:04:11 No.80058 [R]
thats what ive been seeing as well, which is why i just delete the ender files almost every time before i run the script.
Anonymous 05/27/20(Wed)16:12:12 No.81643 [R]
Is there a script similiar to this which can organise the images from each post into folders. Would it be difficult to make, i'm not very good with code
Anonymous 05/27/20(Wed)20:48:58 No.81674 [R]
Does anyone knows if this script has any Git project so i can have it always updated? I've been checking the thread, but don't seem to find something like this.
Anonymous 05/27/20(Wed)23:11:59 No.81718 [R]
as far as i know, this thread is the only source for the script
Anonymous 05/29/20(Fri)08:08:04 No.82041 [R]
I keep getting an error consistently with this creator, any ideas?
Anonymous 05/29/20(Fri)08:10:47 No.82043 [R]
There is a problem in the script that results in file names with spaces at the end which windows doesnt know how to handle .
Anonymous 05/29/20(Fri)11:46:36 No.82084 [R]
After monitoring some more, I can no longer confirm what I've said. It certainly seems like the script tries to do something with existing .part files even if there is an .ender file. Maybe there was just few edge cases that made me think otherwise.
Anyway, the problem is really on server side. I've tried to download a 450 MB file by other means like browsers, download managers and even vps. All failed. I can get the first 60-90 MB, and no more. The way the server handles transfer is weird and unstable. Resume fails most of the times, so starting the download from 0% is more likely to succeed, but in the end chances to get to even 50% of the file are still around zero. I'm guessing the issue is know to admin though. There are also more reports like this: https://yiff.party/bbs/read/81967
Also, found a bug in recent version - pressing a button in the end closes the window, while it should have restarted the script instead.
Anonymous 06/02/20(Tue)14:58:17 No.83402 [R]
It also won't start if you have a space in a folder...
Anonymous 06/02/20(Tue)15:52:24 No.83415 [R]
errrrrrrrrrrrrror downloading
seriously, how do you even get it to properly download something?
Anonymous 06/02/20(Tue)17:32:34 No.83440 [R]
When we get a billion error downloading and have a billion .part files and a file called ender, what are we supposed to do when we get this screen?
I want this to be explained in plain English, step by step
Anonymous 06/02/20(Tue)19:49:01 No.83463 [R]
Anonymous 06/02/20(Tue)20:24:53 No.83470 [R]
what am I supposed to do with ender and .part files? delete it?
Vera !KhDwBGWuMs 06/03/20(Wed)02:43:03 No.83518 [R]
.ender files mark the last downloaded file without errors, or maybe before errors. It's used so the script doesn't have to attempt to download everything, and .part times are just files that failed to fully download.
delete them and retry, what's the worst that could happen.
Nero 06/04/20(Thu)04:13:04 No.83864 [R]
I cant download anything it just keep retrying
Anonymous 06/04/20(Thu)21:30:55 No.84081 [R]
i also get this issue with https://yiff.party/patreon/4339136
Anonymous 06/07/20(Sun)17:50:40 No.84785 [R]
Encountered the same issue with Fantia downloads now. Really had to delete .ender to finish the download of .part files.
Anonymous !cuDH463TvE 06/07/20(Sun)21:16:04 No.84829 [R]
If you encounter a problem downloading a file then please be patient and try again, there's been issues with server now and then. There's no need to delete .ender files for the .part files, they will be resumed. The only reason to delete .ender file is if you have changed your filter list or you need the file (a success downloaded file) back that you have deleted.
Also on the off chance that the .part file might be one byte too much that can cause them to be a partial file for good (yiff.bat will drop the extension ".part" only if their file size and size reported by server are same), but I haven't encountered one. If you think you got one you should delete it and try again.
I haven't encountered one, but I know that yiff.bat is sensitive to the glitching HTML, I would also suggest wait and try again later, after the HTML can be rendered properly.
Anonymous 06/09/20(Tue)02:13:03 No.85066 [R]
trying to download sakimichan's page always gives an offline error, which leads to endless retries. can anyone else confirm this please?
Anonymous 06/09/20(Tue)16:34:59 No.85155 [R]
Running yiff.bat in linux seems to cause error.
python3 -x yiff.bat
and then this error pops up
File "yiff.bat", line 582
if rule := [v for k, v in refererlist.items() if k in url]:
SyntaxError: invalid syntax
any way to fix this?
Anonymous !cuDH463TvE 06/09/20(Tue)17:45:24 No.85165 [R]
Python 3.8 and up. There's Walrus operator (":=") in this.
Anonymous !cuDH463TvE 06/09/20(Tue)19:34:01 No.85183 [R]
DUDE. There's too much comment in that page! Admin should do something about it because it's wasteful and silly.
Anonymous 06/10/20(Wed)00:05:20 No.85223 [R]
so its because the page is so big that the script cant parse the contents?
i was able to download https://yiff.party/patreon/950545 with no issues (apart from the download timeouts)
Anonymous !cuDH463TvE 06/10/20(Wed)00:46:58 No.85230 [R]
Not because yiff.bat can't parse it, it's that page that big... whenever connection drops it'll have to start over. Yiff.party doesn't support resuming on HTMLs. I can see how it can be frustrating when the server or the connection isn't in stable condition.
Anonymous 06/10/20(Wed)13:18:53 No.85341 [R]
guess ill just wait until the servers are running properly again
Anonymous 06/19/20(Fri)00:06:06 No.87485 [R]
is this shit dead yet
cant download anything anymore, always retrying
Anonymous 06/19/20(Fri)18:29:28 No.87686 [R]
links in red arent included in the HTML, is this intended?
link for reference: https://yiff.party/patreon/2584938
Anonymous !cuDH463TvE 06/19/20(Fri)22:52:51 No.87728 [R]
Good find! It's uncommon asset I haven't got chance to work on. It's not part of the body like the attachment files did not. I'll merge it into the body.
Updated yiff.bat, please delete HTML file and shrine.json to rebuild HTML.
I don't know what this will look on actual paysite (Patreon) though, so it cannot be helped. But I will try if creator is also: gay furry artist, does embed url, I'd might pledge :P
Anonymous !cuDH463TvE 06/20/20(Sat)01:49:08 No.87754 [R]
Updated yiff.bat again, this time for embed urls on Patreon
Anonymous 06/25/20(Thu)12:10:17 No.89124 [R]
Is there a way to figure out which yiff can't be parsed correctly or make it so we can set a max time for it to parse before it skips to the next id? I'd run the bat and then it'll hit an ID that just stops. I comment out that one and it does it again at another and it's not like it's going down the list, it would go backwards sometimes. The only thing I can think of to split Yiff into smaller directories and run them in batches but that's kind of backwards if I'm trying to keep a single list.
Anonymous !cuDH463TvE 06/25/20(Thu)14:45:22 No.89141 [R]
It might be hard to place some clue to what creator it's "stuck" on right now while there is parallel downloading going on. There can be stuck connection between yiff.party and computer sometime with seem no timeout (sometime I really have to wait for a while for it to time out), what I found worked (and faster to it) is to disconnect internet from my computer and then put it back on, yiff.bat will immediately throw an offline error, I just tell it to retry. I haven't actually supply timeout amount to there yet, it'll decide when to stop depending on condition of the connection, there's no exact timeout.
Deeper down in creators list can get finished earlier in parallel threads, it would appear to go "backwards sometimes" in the creators list like you're seeing.
Anonymous 06/25/20(Thu)16:58:39 No.89158 [R]
Ok Found my mistake. It was hanging because it got stuck on a YIFF that was deleted. I went through my entire YIFF list commenting out and deleting the ones that aren't available anymore and it's running great staying blue again.
Anonymous !cuDH463TvE 06/25/20(Thu)21:53:32 No.89219 [R]
Oh, right, I should update yiff.bat to skip the deleted creators!
Anonymous 06/30/20(Tue)12:27:34 No.90392 [R]
Is there a way to reuse python seeing as it keeps on restart the .cli?
Anonymous !cuDH463TvE 06/30/20(Tue)12:37:11 No.90395 [R]
Updated yiff.bat support importing from Kemono.party for Fanbox artists, new setting "Kemono.party =" added, HTML server is also improved, small update to the first time setup CLI message displaying a URL to download Python and another direction where to get a 64-bit version.
I can't update yiff.bat to skip yiff.party's "excluded" creators because:
1. Yiff.party actually assigned them as 404.
2. Urllib module the yiff.bat is using to fetch pages/download things will see 404'd page "not found" and 404'd page "creator excluded" same thing.
3. Final say, I'm not thotslobberer so...
Anonymous !cuDH463TvE 06/30/20(Tue)12:39:05 No.90397 [R]
Are you saying it couldn't find Python?
Anonymous 07/03/20(Fri)15:31:05 No.91100 [R]
Thanks for kemono support. Please fix the following... Error text is self explanatory:
> Error downloading: https://kemono.partyhttps://kemono.party/attachments/fanbox/1549213/914289/gs36lwnAmmI81ChsCDGY2Jkm.png
Anonymous 07/03/20(Fri)18:58:57 No.91173 [R]
I'm noticing that R/A key presses stopped working with the latest update. "Restart CLI? Press any key" seems to work now though.
Anonymous !!gYsLYR6/R2 07/04/20(Sat)01:24:53 No.91252 [R]
Sometime it'll only register after two keystroke of a same key, I have wrote about why is this happening >>77213
Unless you mean it really stopped working.
Thanks, it looks like kemono.party sometimes include the "https://kemono.party" in url string and sometime does not. Updated yiff.bat for that.
Anonymous !!gYsLYR6/R2 07/04/20(Sat)01:27:11 No.91254 [R]
Oh, looks like tripcode logic has changed
Anonymous 07/04/20(Sat)10:08:36 No.91315 [R]
Thanks. But it still happens with some images.
> Error downloading: https://kemono.partyhttps://kemono.party/inline/fanbox/bNJGCleMRcuo2u8uYjmUX5dZ.png
Creator ID is 5481606.
Anonymous !!gYsLYR6/R2 07/04/20(Sat)10:27:55 No.91320 [R]
Hory shet! Forgot about this one too. Updated yiff.bat
Anonymous 07/05/20(Sun)09:19:48 No.91587 [R]
Some documentation would improve the experience greatly, but this tool is excellent. Thank you very much.
Anonymous !!gYsLYR6/R2 07/06/20(Mon)07:23:59 No.91875 [R]
The usage already is and should be pretty straightforward. I've updated first time CLI message a little bit and the initial TXT file accommodating new features like artists from different paysites. The other obscure documentations are deep in this thread and I cannot update OP so...
Anonymous 07/08/20(Wed)18:35:29 No.92502 [R]
This tool is amaaaaaaazing! I never dreamed things would go so smoothly! Well done OP! Well done!
Anonymous 07/10/20(Fri)13:18:02 No.92942 [R]
About kemono again. So it works well so far but I was thinking what could be done about the ordering. See, since there are some hashes instead of original file names, the script downloads them without any indication that would help with ordering. I thought about asking kemono about it, but original imported pages on kemono are actually some html files that include images in their original supposed order. I think something can be done about this. E.g. when downloading 745537.7Y540DiosJ7IEp9cKse3isXq.jpeg, note its position in the html source and name the file accordingly, e.g. 745537.0003.7Y540DiosJ7IEp9cKse3isXq.jpeg. It may also be possible to rename already downloaded files accordingly if they are detected.
What do you think?
Anonymous !!gYsLYR6/R2 07/12/20(Sun)18:50:20 No.93457 [R]
Number increment is vague way to order files and can be messed up since Fanbox allows editing post to add or delete files, or maybe when bootleg site has the missing files/wrong order, etc. We could capture the secondary ID as a way to put files in right order. I'd rather not incorporate this into my script right now if that means to enforce new filename into our existing collection.
HTML builder will have them in order the way they were originally.
Anonymous !!gYsLYR6/R2 07/15/20(Wed)12:44:38 No.94000 [R]
Updated yiff.bat for custom dir
Obviously useful for yiff.party's shared files - add these to yiff.txt:
> Shared files/* for https://data.yiff.party/shared_data/
> Shared files/
Explanation: First rule will attempt to save files to "Shared files/" from https://data.yiff.party/shared_data/, it's valid only if this dir exists, second rule to allow creating this dir.
> Update me/* for https://
Can catch outlaw domains for attempting to save to "Update me/" that didn't exist, useful to catch old rules because of domain changes.
Anonymous 07/16/20(Thu)19:11:11 No.94266 [R]
Thank you for the new Patrol feature, very useful. However, could you please have it output the list of corrupted files into its own file?
Also, the archive testing feature, I was not able to get it to work. It tested some archives just fine, but eventually stalled in the background. I think 7z must've given it an output it wasn't expecting so it just stopped. Both 7z and python had 0 activity going whatsoever for like, an hour.
Lastly, even though yiff.bat says it's rare, it turns out that the artists I follow update their already posted art with corrections quite a lot, or in some cases, put in a different background, or change something about the picture that the person who commissioned them didn't like (but perhaps I did like). I can't imagine not using collisionisreal + editisreal now, it's odd to me that they're considered features only for debugging.
Anonymous 07/16/20(Thu)19:13:40 No.94268 [R]
Oh, and for the images that are corrupted on disk, it would help if you printed out the full file paths. It would make it easy for me to make a quick script to send them all to the recycle bin.
Anonymous 07/16/20(Thu)19:16:43 No.94269 [R]
Ugh, thought of one more thing. It would be nice if I could have Patrol only test archives and not images. I already tested all my images today, so if and when the 7z functionality is fixed, I'd like to be able to test my archives without having to re-test my images.
Anonymous 07/16/20(Thu)19:18:52 No.94271 [R]
...also, thank you for your hard work, lol
Anonymous 07/16/20(Thu)21:46:23 No.94304 [R]
Patrol is causing some odd warnings with high-resolution images, I'm not sure if it's interfering with its ability to discern whether an image is corrupt or not:
lib\site-packages\PIL\Image.py:2797: DecompressionBombWarning: Image size (100000000 pixels) exceeds limit of 89478485 pixels, could be decompression bomb DOS attack.
Strangely there is nothing proceeding that last parenthese, that's the whole error.
Anonymous 07/16/20(Thu)22:05:52 No.94309 [R]
Fixed it by adding: Image.MAX_IMAGE_PIXELS = 2000000000
after the line: from PIL import Image
Anonymous 07/16/20(Thu)23:16:10 No.94324 [R]
I think Patrol should run after an artist is retrieved, rather than before, that way if any of the files that were downloaded are corrupt, they're marked on that run rather than having to do an extra run with Patrol on. This is made more irritating by the fact that if there's no new images, you don't get the new yiff.html file that tells you what's corrupted.
Anonymous !!gYsLYR6/R2 07/17/20(Fri)01:16:21 No.94342 [R]
Added new setting "Patrol mediocre =", out of debug menu! Will scan files after downloads finish and will save result to HTML even if there's no pictures.
Scary number, would 200000000 be fine? Added this number to yiff.bat
Well, I think I've made enough changes to yiff.bat for today based on feedback here. I'll read everything & talk later.
Anonymous 07/17/20(Fri)03:47:19 No.94363 [R]
200000000 is definitely not enough. It was tripping up on a 12190x17505 resolution file, and that's a pixel count of 213385950. One of the google results I came across on this problem was dealing with an image of about 300000000. I would say put it at least at 400000000, perhaps 500000000 just to be safe.
Anonymous !!gYsLYR6/R2 07/17/20(Fri)13:51:20 No.94420 [R]
Updated yiff.bat with 400000000 pixels allowance, and also now you can put in the path to 7-Zip to patrol setting for archive scan support.
I'm not sure if I'd like to try enable 7-Zip while disabling image scan feature of the patrol setting, both of these are supposed to be a limited run. Ideally only patrol once in a while. Maybe image scan feature was there to discourage you from doing constant archive scans? :D
I wonder what could be causing 7-Zip to hang on your on some archive files, can I see it for myself? Is it random?
Anonymous !!gYsLYR6/R2 07/17/20(Fri)14:25:36 No.94426 [R]
And I'm not really sure if I can take the collisionisreal and editisreal out of debug menu. I mean, I have them enabled in patreon.bat (yiff.bat clone) for \patreon\ folder for people I actually pledge. But I'd imagine most people just use yiff.bat to download shit off from yiff.party where editisreal is useless for it anyway. Meanwhile collisionisreal, idk
Anonymous 07/18/20(Sat)02:47:14 No.94538 [R]
collisionisreal must be what's giving me all the edited versions of pictures then, because I currently only use it for yiff.party
Anonymous 07/18/20(Sat)02:48:31 No.94539 [R]
> I wonder what could be causing 7-Zip to hang on your on some archive files, can I see it for myself?
Well it certainly would've helped to have a setting that let me do only 7-Zip scans without image scans so I could debug it quicker...
Anonymous !!gYsLYR6/R2 07/18/20(Sat)21:55:33 No.94709 [R]
It's got lot of false positives, sometime inferior copies, would be good idea to leave in debug menu for now. I thought it's legit until I found another picture in different name, got me for a second. The duplication is usually in misleading filenames. Has there been a legit picture you'd only have when collisionisreal is turned on?
If you want to "debug" something you can always edit anything in batch file. Maybe try blank out the array in variable "imagefile" in batch file.
Anonymous !!gYsLYR6/R2 07/22/20(Wed)22:47:33 No.95714 [R]
Updated yiff.bat for more download statuses (and upload status during HTML server). Something new in window title! It's using 256 braille characters like a binary counter representing progress of a file downloading, 256 flops to completion. There will be 8 of it in window title based on number of available slots for parallel downloading.
Anonymous 07/23/20(Thu)13:22:03 No.95824 [R]
Thanks for the update
Can you give an update next time to know which page is causing the offline status to appear?
Anonymous 07/25/20(Sat)08:53:20 No.96227 [R]
I don't know if something changed with the script or with yiff.party, but the retry count has been skyrocketing. Is the timeout too low? Maybe it would be better to stop forcing a=post_view_limit&d=all and instead retrieve page by page?
Anonymous 07/25/20(Sat)10:59:21 No.96237 [R]
I figured out why the 7z scanning was freezing. Apparently it's when an archive is password protected, it stops and prompts you to enter a password.
This means a function is going to have to be made to read the stdout/stderr of 7z instead of just getting the error code.
Alternatively, you could use this library: https://github.com/miurahr/py7zr
Anonymous 07/26/20(Sun)18:53:47 No.96611 [R]
Seems like yiff.bat fails to download files that are under the same patreon post number, and that have exact matching file names. I've seen some posts that have multiple 01.png files in them, but yiff can only get one of them
Anonymous 07/29/20(Wed)03:06:20 No.97140 [R]
Wheres the download link for the latest version?
Anonymous 08/01/20(Sat)05:22:44 No.97936 [R]
wont start bro i just get this error
New ID! I'll check creators list from yiff.party . . .
Traceback (most recent call last):
File "F:\Python\yiff.bat", line 1537, in <module>
FAVORITE, rules = finduser(rules, FAVORITE, id=CREATOR_ID, stderr=f" Bad ID!
Please remove or comment out {CREATOR_ID} from {rulefile} then restart CLI.")
File "F:\Python\yiff.bat", line 1477, in finduser
userlist[0] = json.loads(get('https://yiff.party/json/creators.json', requir
File "F:\Python\yiff.bat", line 754, in get
echoMBs(threadn, Bytes, int(dl/total*256) if total else 0)
File "F:\Python\yiff.bat", line 329, in echoMBs
fx[0][x if threadn else 0] = fp[ff]
NameError: name 'fx' is not defined
Anonymous !!gYsLYR6/R2 08/01/20(Sat)07:25:50 No.97947 [R]
I could try make it tell what url that caused it and maybe abilities to skip it.
yiff.party always has been unstable at random times, the cookie is the first thing yiff.bat do when contacting yiff.party
Fixed! Sorry, it's a new feature without enough testing.
"collisionisreal = True" in batch file can help, is it an actually missing picture that we couldn't get it other than turning on the collisionisreal?
I updated yiff.bat to do about password protected archive to be seen as corrupted (clever trick here https://sourceforge.net/p/sevenzip/discussion/45798/thread/2b98fd92/), untested.
Anonymous 08/13/20(Thu)14:30:29 No.101604 [R]
Is there any way to make it so it doesn't parse all the creators in yiff.txt all at once first before starting downloads? I know I can do it one by one manually but was just wondering.
Anonymous !!gYsLYR6/R2 08/13/20(Thu)22:56:06 No.101764 [R]
Not until I updated yiff.bat just now, this reminded me that there should be separator in yiff.txt when to queue next favorites. There's "end" that has been implemented for a while already used to stop reading from favorite list.
"then" is a new separator! Write in yiff.txt will execute recent favorites in list then resume when done.
Thank you for reminding, this feature is kind of overdue.
Anonymous 08/13/20(Thu)23:32:53 No.101767 [R]
I have a feature request for the yiff.party scraper. Could you make it so that it'll just elegantly skip over failed lookups or non-existent IDs? I often like to make a list of artists that haven't been added to yiff.party yet and check the site every once in a while to see if that's changed.
(I don't personally use the lookup since that doesn't work well with setting a custom directory, so I grab their creator ID from their patreon page by searching the source code, and then I add it ahead of time but commented out until I find out they've been added.)
Anonymous 08/15/20(Sat)11:32:29 No.102095 [R]
hello, does anybody know how to download url text only for specific creator ? my internet is super laggy and its failing every file that i download even for 1 queue, i planned to download it on idm
Anonymous 08/15/20(Sat)11:38:55 No.102096 [R]
What actually is this? What does the batch file do???
Anonymous 08/15/20(Sat)11:42:25 No.102097 [R]
its downloading, but always failed. i know the batch will retry at the next loop to resume the failing file. alright, i change my question. how to loop the batch file without pressing any key ? it will always loop, but at least i don't have to press any button
Anonymous 08/28/20(Fri)20:25:22 No.106431 [R]
Please support more than just Fanbox on Kemono.party, there is Patreon content that I would like to download on there that is not available on yiff.party.
Anonymous 08/29/20(Sat)17:30:28 No.106776 [R]
How do I use this with kemono? Whenever I paste kemono id in, it just say bad id and exits.
Anonymous 08/29/20(Sat)17:44:47 No.106788 [R]
Anonymous 08/29/20(Sat)22:20:06 No.106895 [R]
how do i use it with fantia
Anonymous 08/31/20(Mon)18:32:12 No.107434 [R]
Hi, just started using this today to archive stuff after being spooked by the recent downtimes, it seems this doesn't recognize symbols like plus signs in the names of files. I've been downloading Eltonel's stuff (https://yiff.party/patreon/393249) and some of his posts have different files named like File, File+, File++ and so on that all get interpreted as just 'File' by yiff.get. As a result it skipped over the apparent extras until I turned on collisionisreal, which managed to grab those and rename them as File, File (2), File (2) (3), etc. Not exactly an elegant naming convention, might get really long if there happens to be a post with a ton of files named this way, and it replaces the old file names. This is probably a low priority issue and I don't know if OP will even see this, but I thought I'd bring it up here just in case you decide to do something about it.
Anonymous 08/31/20(Mon)18:53:03 No.107450 [R]
Update: okay apparently I posted this too soon, it's actually inconsistent between posts. For instance on https://yiff.party/patreon/393249?p=9 the post "Tris" had none of its + symbols recognized by yiff.bat, it simply deleted them from the file names, so the post file and first 2 attached files all got named according to File, File (2), File (2) (3) as noted above, and the other 3 still had different names after the +s were deleted. The same applies to the 3 versions of vivgilda_shower on the next post "Mixed bag of a lot of things ver. 2: Electric Bogaloo." After that on "Marcille -wip-" it leaves the ++ and ++++ versions alone, and also recognized that the post file and the attached file, both called 'marcille', were different after I enabled collisionisreal. Then a few posts down on "Poppy Bukkake" it again recognizes the + signs, and this post's files all downloaded properly the first time even without collisionisreal enabled. The inconsistency is really weird and it seriously bothers me.
Anonymous 08/31/20(Mon)19:05:04 No.107464 [R]
Update to the update: I realized what's going on, it seems the files are actually named inconsistently within yiff.party's database for whatever reason which you can see from clicking on the direct links to the images, so that in some cases the files listed as having +s in their names in the post actually share the exact same name in the database and lack the symbol characters while others actually have said symbols as part of their filename in the database. I can only assume the file naming discrepancy is due to them coming from different sources.
Probably no need to deal with this any more, I was just worried that the inconsistency was due to borked code in yiff.bat which might cause problems somewhere down the line, but this is entirely due to inconsistent filenames on yiff.party so it should be fine.
anonymous 09/08/20(Tue)03:02:05 No.110044 [R]
Would you consider putting a link to this thread in your batch file, found the mediafire link on another site and this thread is usually buried quite a bit, you could also put your tripcode as well.
Thanks for your hard work!
Anonymous 09/10/20(Thu)01:24:50 No.110696 [R]
Went to open yiff.bat today and it immediately closed on me. Was able to get a screenshot of what it was saying before it was closing. Tried repairing Python 3.8.5, which didn't work. Mind helping me out?
Anonymous 09/10/20(Thu)01:32:35 No.110705 [R]
Also, I was using this well before today. Have an external HDD dedicated to it.
Anonymous 09/11/20(Fri)00:54:48 No.111083 [R]
Please fork gallery-dl and implement this https://github.com/mikf/gallery-dl/issues/315
Anonymous 09/11/20(Fri)05:04:05 No.111135 [R]
Hi, if I want to use the updated version, is it fine to just replace the existing yiff.bat file in my folder? Also, is anyone else getting the "Error getting new page for #########" message? I assume it might be due to temporary connection issues with the site today but not sure.
Anonymous 09/11/20(Fri)13:52:53 No.111246 [R]
You SHOULD be fine. That's what I went to do 2 days ago and it busted on me. Could just be me though. If you want to update, put the new .bat file in a folder and see if it works before you replace it. If it does, go for it. If not, then stay on the version you have already.
Anonymous 09/11/20(Fri)23:42:26 No.111392 [R]
Dont worry about this issue anymore, I got it (somewhat) functional. Had to add what I entered in the image. (It still shows an error but it downloads the files I want, so Im happy.)
Anonymous 09/16/20(Wed)03:09:55 No.113005 [R]
yiff.bat's HTML pages have a handy 'Links in this HTML' button, which is nice, but the problem is URLs in a yiff.party post's Embed data aren't copied into the HTML, which often link to things like Google Drive or MEGA - only the URLs in the descriptions.
Anonymous !!gYsLYR6/R2 09/16/20(Wed)15:24:19 No.113164 [R]
What creator? Yiff.bat should append embed data below the main post body becoming part of it.
Anonymous !!gYsLYR6/R2 09/16/20(Wed)15:33:00 No.113167 [R]
Oh, I think you'll need to delete shrine.json if you have been using yiff.bat for long time, older yiff.bat doesn't append embed data below body.
Anonymous 09/17/20(Thu)19:17:57 No.113652 [R]
That didn't work, unfortunately. Deleted all the shrine.json files, ran yiff.bat again, and the issue is the same.
Here's an example, I picked a creator with not too many posts. The Google Drive URLs (which are dead now, but that's unrelated) in the embed data on the last 3 posts on the last page are not captured by yiff.bat: https://yiff.party/patreon/10822463
'HD Image Archive', 'HD Alt Archive', and 'PSD Archive' to be specific. Here's a screenshot: https://i.vgy.me/qgN1Pg.png
If it were working properly, these URLs would be included somewhere on the page:
Anonymous !!gYsLYR6/R2 09/17/20(Thu)22:08:27 No.113710 [R]
Sorry, forgot one more step, you will also need to delete either Ender or HTML to force rebuild HTML.
I tried yo-lander and the embed data did appeared below body for me so I know it's working.
Anonymous 09/22/20(Tue)14:45:29 No.115180 [R]
Hey op, I think that yiff.party is under the very real threat of going under due to a lack of funds. Can you please mirror your tool over to github?
I would like Yiff to be mirrored because if yiff.party does go down this tool can still be used on other sites like Patreon and Kemono. It would be a shame if this tool went down with this site.
Anonymous !!gYsLYR6/R2 09/22/20(Tue)18:23:51 No.115217 [R]
Ha, yiff.party can only go down with a link to yiff.bat, I'll link it elsewhere.
Anonymous 09/24/20(Thu)06:13:54 No.115712 [R]
getting this error today OP
Connecting to Yiff.Party . . .
New ID! I'll check creators list from yiff.party . . .
Thread 0 was killed because the finished receiving data doesn't match header's t
Restart CLI? Press any key to continue . . .
Anonymous 09/24/20(Thu)15:39:03 No.115815 [R]