Website Redesign ...

  • Thread starter Thread starter K-dub
  • Start date Start date
K

K-dub

Well-known member
I've not touched my music page in months, but I recently discovered a way to allow free downloads w/o the web bots being able to harvest downloads ... which is why I had to previously require registration before allowing folks to download.

I'm in process of major overhaul ... so please feel free to check it out and download away for free.

Just sharing stuff ... so feel free to ask questions. This is the sharing/listening place. I'm happy to answer with what I know.

Kev's Neglected Place

Kev-
 
dobro said:
What's the thing about the web bots?
Click to expand...

I don't know what his was, but on a site I manage there's a player called NSPlayer that has been sucking bandwidth by streaming mp3 files by the hundreds. It's mostly used in China. I simply blocked NSPlayer with the the .httaccess file. That stopped a bunch but max transfer was being exceeded by other means so I blocked a selection of the most abusive IP address ranges, also using the .httaccess file.

It looks like K-dub used a form button to hide the mp3 files so they couldn't be directly accessed.
 
Boulder is exactly right, Paul. Web bots scour the net under the radar looking for opportunities. They'll eat up site bandwidth to extremes, if allowed. They even sign up for a bbs and then post spam. For instance, one found my file "Overtime", attached itself and linked back to this board from it ... signing up as a member and then posted one liners like "Great!" when all it was was an attempt to sell some chiropractic service or nonsense like that. Eventually I had to close the thread due to the bot activity.

Consequently, I had to initiate protection against the bots by requiring registration to the site ... which requires a captcha prompt to prove human, before allowing anyone to "see" the download links. As Boulder correctly guessed, I found another way of allowing free downloads by hiding the download links behind a button the bots can't read. I can't take credit for it. It's a program written by someone else I found as a site add on.

As a matter of note, one way of protecting was mentioned by Boulder: blocking access using the .httaccess file ... which requires a password to continue. Another way of hiding things is by changing the location of the root folder where the files are stored away from the domain name where the links are. What this means is that if the original storage folder was in Http/hostname/domainname/musicfoldername/filenames -- you'd simply remove them from the domain name path and put them outside of it ... resulting in: http://hostname/musicfolder/filenames. It's just another layer of protection against bot bandwidth drain. Of course using both methods is possible and adds yet another layer of protection.

What I'm going to be adding from here for function is a way to both preview the tunes with another button, and purchase hi res files for a nickel or so ... plus figuring out a way to provide free download for the whole album (right now it's just individual songs) and of course, eventually making everything look nicer by cleaning up the front page.
 
If I could also ask for some help from anyone who visits ...

I'm doing a lot of the coding/design by hand -- so if something doesn't work right ... say the wrong song got downloaded or played ... could you give me a heads up? My personal contact info is right on the site -- and THANKS!

Kev-
 
K-dub said:
As a matter of note, one way of protecting was mentioned by Boulder: blocking access using the .httaccess file ... which requires a password to continue.
Click to expand...

You can do more than require a password. I blocked the NSPlayer and the IP ranges outright. If you leave off the last number of an IP address it applies to that whole network.
 
Interesting information. Thanks for the clarification!

Did you write the .htaccess file yourself or did you use a program to do it for you? I pretty much know what it does, but I've never done it myself. I was wondering if it might be a useful layer of extra protection ...
 
K-dub said:
Interesting information. Thanks for the clarification!

Did you write the .htaccess file yourself or did you use a program to do it for you? I pretty much know what it does, but I've never done it myself. I was wondering if it might be a useful layer of extra protection ...
Click to expand...

Although I've dabbled in writing my own I got help with these. For no particular reason one is in the root directory and one is in the audio directory. The one in the root was generated by a tool in the hosting service and the one in audio was something I got from a search.

Root:

Order Allow,Deny
Allow from all
Deny from 123.171.1.
Deny from 123.125.67.
Deny from 220.181.51.
Deny from 110.7.186.
Deny from 123.121.114.
Deny from 119.182.38.
Deny from 124.132.119.
Deny from 218.59.184.
Deny from 117.136.9.
Deny from 112.225.77.
Deny from 113.124.144.

Audio:

SetEnvIf User-Agent MLBot BAD_BOT

SetEnvIf User-Agent NSPlayer BAD_BOT

Order Allow,Deny
Allow from all
Deny from env=BAD_BOT
 
Very interesting -- so you didn't password protect, you just denied surgically based on the linux script.

I don't know where those IPs are, but I'm guessing most of Asia and Russia got axed? :)
 
K-dub said:
Very interesting -- so you didn't password protect, you just denied surgically based on the linux script.

I don't know where those IPs are, but I'm guessing most of Asia and Russia got axed? :)
Click to expand...

The numbers in the first one are networks I guess. The last number makes it a particular address, but I wanted to block whole ranges of addresses. Really, I should put those just on the audio directory and let them surf the rest of the site.

The networks I blocked are mostly big cities in China. I just scanned through the access logs and copied/pasted ones I saw a lot of.
 
My folders are in line down from the domain name still, and there has been a lot of bot activity already ... even since I uploaded the new mp3 files on Saturday. Interestingly, I already had some mp3s in another folder down from the domain that I had deleted as I uploaded to the new folders accessed by the free download buttons -- and I had almost no bot activity before..

The front end indicates a lot of guests, but the back end seems to be always down one or two counts from the front end -- which I'm attributing to bots. Hard to say.

I should probably check server stats and see the IP addresses of origin. Or I could do what you did, insert the .htaccess file and eliminate certain bad bots from being able to find.
 
K-dub said:
I've not touched my music page in months, but I recently discovered a way to allow free downloads w/o the web bots being able to harvest downloads ... which is why I had to previously require registration before allowing folks to download.

I'm in process of major overhaul ... so please feel free to check it out and download away for free.

Just sharing stuff ... so feel free to ask questions. This is the sharing/listening place. I'm happy to answer with what I know.

Kev's Neglected Place

Kev-
Click to expand...

Cool site, Kev. Lot of stuff on there.
 
You must log in or register to reply here.
Back
Top