Calling Dexter J, I need to e-mail you the Saabvideo logs

Dexter J,

I need to e-mail you the Saabvideo logs. I just received them from the system administrator in what seems to be a .gz compressed archive.

SG

Reply to
Saab Guy
Loading thread data ...

Actually - all you need is to pick up compression tool for the file. GZ is a type of zip file that should have your logs inside. Have a poke around here for an app to decompress:

formatting link

Once they are decompressed, use a text editor (word wrap off) or spread sheet to review transactions with the server. Here is a fairly good start page regarding how read raw Apache Logs:

formatting link
.. don't worry about the server set-up detail outlined above just yet, you just want to get a handle on visit patterns and from what IP ranges. There is some good detail above on what each item in a given log line means as you scroll down a bit.

Most folks wimp out and go for a 'web log analyzer' - I don't recommend them until you at least have enough basic literacy to understand the raw log. The features on most log analyzers make *much* more sense if you understand what they are doing for you.

So, very basically, each visit should generate several log lines as each page item is passed to the visitor. You are looking for anomalies in the pattern where the visiting IP *doesn't* read the top page and all it's images first - and/or - repeated visits from the same IP on any kind of schedule - but most importantly - video downloads without any corresponding top page visit.

Towards the end of each log entry the 'user agent' is listed. You should see visits from google, yahoo et al (relatively useful bots/slurpers). They usually only take the page text body - but not always. Pay attention to the detail here, because I'm still betting you are seeing a pile of visits from things like Wget, DA, psbot and a bunch of others. Here is a good page for reviewing bots and their place in your universe:

formatting link
Finally - as to tightening up your site security after you figure out what is going on - that unfortunately is where we get back to the question of my per diem and your personal responsibilities as a 'webmaster'.

As I have said previously, I don't think it's worth your money to hire me for what is simply a hobby site on a shared server. I dearly wish you had a fat sponsor and it was worth the cash because I like the idea of your site, but I guess the best you are going to be able to do is enough reading to stop the obvious packet leaks like most everyone else.

I will again repeat my advise as regards your basic architecture though.

You should at least look at a teaser index page with several video specific sub-pages as your top page now runs better than 1.2 megs every time it's viewed without downloading any videos. While you are at it - you might want to look at changing links to specific video's you have already posted out here - it will help you narrow down who is swiping your vids (they will now appear in the error logs as dead link attempts) and it should slow down the transfer totals for a while.

Reply to
Dexter J

Dexter J wrote: While you are at it - you

I don't know much html, and no Apache, so this is a dumb question, but would there be an easy way for Ryan to eliminate the href on the thumbnail and replace it with a "download" button to keep people from just clicking and viewing live or casually viewing that way? I'm thinking he could force users to save it to view it, and to hide the videos from web bots.

Reply to
saabyurk

Yes - he could drop the vids into a couple of passworded sub-directories on the site - then provide a password hint in the message box that pops up when a user clicks through to download. Then make an adjustment to the mime types to attempt to force an application.vnd exchange to a known null client.

Not fool proof by any means and not everyone 'gets it' when they come across the password hint in the logon box - but it does force a somewhat more targeted automated slurp and mostly forces a download (this latter part being driven much more by the particular browers visiting). Do not include reference to the password and ID in the page link other than to say there is one and how to figure it out.

It's only a stop-gap measure though, pro-slurpers are clever beans. The correct way to go right now is to use mod-rewrite and a cgi so that the system demands several parameters be met in addition to hitting the link and filling out the password.

Someone is going to suggest an FTP server somewhere this process. I

*highly* recommend *against* it at this point in time and have dropped the standalone FTP server as part of the Mark-III build here at Radio Free Dexterdyne.

There are several very serious things going on regarding the FTP cracking and the firewall worlds right now that is rapidly making FTP server technology both unsafe and unreliable in production. I built a backend cgi that performs the all same functions as FTP that, while not as easily load-balanced as pure FTP, is *substantially* less open to exploit and more reliable through most firewalls.

If you do go with the hinted passworded/mime type .htaccess kit - you want to cycle the passwords every month or two. This is sort of labour intensive on a bigger site. Not that much of a hassle for a small hobbie site though.

Still - at the end of the say - it's all in the log work though. Know what's supposed to happen there in a normal exchange and it makes it

*much* easier to deal with exceptions.
Reply to
Dexter J

Dexter,

I don't have the time now to read the logs, I thought you were offering your time to help given that I have spent approximately 280+ hours to create

formatting link
for each respective video for you folks. :)

SG

Reply to
Saab Guy

time to help given that I have spent approximately 280+ hours to create

formatting link
for each respective video for you folks. :)

You know what? You're a real pain in the arse. Someone tries to help you and all you can do is come back with is more requests. You should really try to understand that

formatting link
is your hobby, and when one of us takes the time to try and help you with a particular issue, it doesn't mean we want in on the project.

Reply to
Grunff

time to help given that I have spent approximately 280+ hours to create

formatting link
for each respective video for you folks. :)

Grunff,

I just thought I was being nice. :)

Sheesh Grunff. :)

SG

Reply to
Saab Guy

Salutations:

While I don't think I would be quiet as forthright as brother Grunff, I feel that I should point out that at 500g a month, you don't really have time *not* to look at your logs.

I suppose it is of very little passing interest to most normal people like yourself with real lives more than 25 feet away from a stinking ethernet jack - but I have better than four years into Radio Free Dexterdyne (my hobby solution) and all the hoopy things it turns out it can do.

As you don't know me from Adam, I feel I should also tell you that I've actually been at this i-net thing since the late 80's here in Canada when it was still a 300 baud TTL University network us geeks used to pass French Postcards around on between systems dumps for the chem labs. I'm not trying to brag or anything - but I want to set some context for the comments below.

I fear sir that you are making a mistake that *so many* good basic ideas for websites make early on. While I may be dead wrong - I'm pretty sure that less than 30% of your total transfer represents real human beings in front of a computer at your actual site.

Don't feel bad though - that was what the whole 'internet bubble' was really all about anyway.

So what advise I have provided, which I'm afraid you pretty much have all of until you can make some time to analyze your own logs and come back with some deeper questions not normally available through google, is as posted here so far.

Now, all that said, please don't feel I'm trying to be rude or that I am being dismissive of your work. More than most people, I know exactly how tough (and expensive) it is to get actual honest new content online in the any context (hobby or professional).

Which is exactly why I think your content is being piked by the pirates, most of whom have much better advertising sales staff than you and why I have been trying to give you some real leads before you throw up your hands in disgust and walk away from it all.

All this said - my grandfather never allowed a radio or television in the house on the deeply held premise that furniture shouldn't speak or sing. He lived to be 75 on a pack of filterless Craven-A and a pint of Gin a day all his adult life.

This internet thing isn't for everyone despite, as I previously mentioned, the common wisdom that any 6 year can slap up a full on Multi-Media portal and become the next iTunes without ever really trying.

Sadly, like most things in life, that is just no longer true - because several of the 6 year olds have figured out how to hijack other people's content before the site owners can ever make a profit on it themselves.

Reply to
Dexter J

Dexter,

That is all good news. I will pass this along to the sys admin for the site :)

Thank you! :) I do appreciate your time and help.

SG

Reply to
Saab Guy

You know me Dex, especially after a hard day at the office ;-)

Reply to
Grunff

time to help given that I have spent approximately 280+ hours to create

formatting link
for each respective video for you folks. :)

Ayup.

Reply to
Dave Hinz

Me too brother Grunff - me too.

Again, my grandfather lived to be 75 on a pack of Craven A filterless and a pint of Gin daily and did not allow a Radio or a television in the house. He was a QC here in Nova Scotia and so never spent any real time at exercise.

So - in the face of the obvious life shortening aspects associated with the former - I sometimes wonder if he may have known a fair bit more about technology than you and I given the latter.

Cheers brother Grunff.

Reply to
Dexter J

Bro Dex,

You know, I have considered this life philosophy stuff at great lengths, even before your mentioning it. I am convinced, though not solely through your evidence, that human kind is by far its own worst enemy. It surely seems that a simple but pleasurable existence is the shortest route to fulfillment. But we insist on cluttering our lives up with all sorts of unnecessary detritus.

As I approach the mid century mark in my life I am leaning further toward the attitude of living for the moment. I just picked up a fine motorbike, a naked Ducati 750. Hell yeah, I figure if I get creamed out on the road by some "brain-dead booger-head", I already have good (enough) insurance and my kids are already college educated and will prosper. In the meanwhile, I think I'll do whatever damn well pleases me and get on with what remains of my mortal existence. Especially since I remain somewhat uncommitted as to anything beyond it...

Does anyone else find even the least bit of irony in the fact that most old folks (yeah, I'm talking older than me) are far more health conscious than the 20 somethings?

How any of these previous rambling relates to the tedious whining of Mr SaabVideo in this thread is a stretch at best. Sorry... Now back to the Saab Video Channel...

Reply to
Malt_Hound

your time to help given that I have spent approximately 280+ hours to create

formatting link
for each respective video for you folks. :)

Dave,

No, I was just being nice to you folks, but barely anyone thanked me for it.

That's ridiculous. Those who have downloaded the movies and never thanked me, should feel a bit guilty.

SG

Reply to
Saab Guy

your time to help given that I have spent approximately 280+ hours to create

formatting link
for each respective video for you folks. :)

should feel a bit guilty.

Saab Guy,

No, you should be feeling guilty for commandeering and dominating this group. Your previous behavior shows you are highly unlikely to do this, but perhaps your traffic is sufficient to justify creating a couple of new groups:

alt.videos.saab For your periodic posting notices, requests for binaries, and IT consulting inquiries

alt.fan.saab-guy For your frequent, pouty demands for ego massages and other miscellaneous approval-seeking

There are many useful, fun, and/or interesting Web sites relating to Saabs. Few of the webmasters of such sites post here demanding cash contributions, free technical services, and bronze statues built in their honor. So I'm afraid all I can say is, "Saab Guy, you're doing a heck of a job."

_________________________________________________________________ Messages sent to the address in the header will be ignored. Personal replies should be directed to user k.l.bailey at the Institute of Electrical and Electronics Engineers.

Reply to
klbailey_usenet

your time to help given that I have spent approximately 280+ hours to create

formatting link
for each respective video for you folks. :)

should feel a bit guilty.

Why should anyone feel any *GUILT* ! ?

It was *your* project. No-one asked you to do it. It's *your* baby.

Some posters here have sensibly suggested you reduce the file sizes too and you've ignored them. No surprise then that the site is a bandwidth hog. Although I personally have no intention of downloading 2.5G of video just to see a promo for example.

Now get over it.

Graham

Reply to
Pooh Bear

Ok, I was going to leave this but no, you seem on a mission to be praised for something that perhaps wasn't even asked for. If you choose to embark on a mission spending your time and money and someone doesn't thank you for it then you have to ask if they wanted it?

Just because someone has downloaded a video doesn't actually mean they

*wanted* you to do it, just that it was there, novelty factor, lets have a look, bit of fun.

Everytime someone hits my webpage, they don't send me an email, thanking me for the information, every time they download something, they don't send me an email thanking me for making it available. That's just not the way the web works.

should feel a bit guilty.

Why? Sorry dude, you spent your time and "offered" it to the world, don't expect everyone to fall over themselves thanking you!

FWIW, the start page takes way too long to load because you've got thumbnail after thumbnail and you've already been told about this.

The size of the videos is such that I can't be bothered to download more than the one that I did. That one took several hours to arrive and I am just not that excited about it. If they were much smaller then fine but as I have said before, for me, I'll just watch it on the PC, no way am I going to spend more time authoring a DVD in order to watch an advert, I have enough of those already beamed at me all day.

You have been offered various solutions about how to improve things, Dexter has already commented on the home page, you've had suggestions about a thumbmail and maybe a sample clip which then links to a bittorrent source...

...BUT YOU'RE STILL NOT HAPPY!!!!

You now want people to thank you every time they download a file?

I give up.

David.

Reply to
David Taylor

should feel a bit guilty.

Yes, YES, we understand. You put up a website with questionably legal downloads of material you clearly don't own. I'm sure it's a very _nice_ website of questionably legal download files.

Then you whined about hosting cost, and several people suggested that the numbers you were mentioning were way high. You ignored us.

Then you whined about bandwidth cost, and several people suggested bittorrent. You ignored them.

Then you whined about weblogs, and at least one person spent a non-trivial amount of time giving you free advice on how to use them to identify your bandwidth problem. Instead of taking that advice and using it yourself, apparently you feel that "Hey, this guy is good for free consulting; he knows how to do it, let _him_ do all the work". I recognized the pattern early on, and 20 years ago I had already decided that that wasn't a game I'd play any more.

(snip)

He'd be paying less for hosting if he had taken the free advice offered. He'd be paying less for bandwidth if he had done any of _several_ pieces of advice freely offered.

He'd know how many people are enjoying his site if he'd look at his freaking logs. News flash: website visitors rarely send fan mail. If it sucks and they hate it, they just don't come back. Hits is all there is, bucko.

Dave Hinz

Reply to
Dave Hinz

Salutations:

OK brothers - I think I'm more than a little uncomfortable with the flaming SG is taking in a thread with my nick on it. Deserved or not.

I feel, though I guess his site is sort of touchy point for many here, that this thread is still not entirely in the best tradition of this particular NewsGroup - one of the best on the i-net in my books, as I have said on many occasions.

So, the greater points having been well made about netiquette, I sort of feel that I should comment in defence of SG that everyone is a newbie at something sometime. That is why I offered the advise in the first place.

Brothers - I really do find the idea of a site dedicated to archival advertising footage from SAAB sort of interesting. Sort of like the Prelinger archives if anyone has time to visit there. As it remains sponsor free - all I was trying to offer were some leads on how better to operate it so it doesn't disappear from the firmament.

I happen to have one last, hopefully thread ending, link to offer with my apols that it's on my own hobby site.

I try to not offer links to my own site other than Top Tune o'be-do-da-day in my sig for a lot of good reasons - not the least of which being that it always feels like I'm pitching business with my electro-pals here. Which is always a black-balling offence in my books too and why you are taking such a beating here regarding your plea for help as regards saabvideos.

So - SG - grap yourself a big coffee and have a careful read over this:

The Village Bicycle - Basic HTML

formatting link
It's a long (what a surprise huh?), general and very basic primer on putting up a hobby website that I really feel you might benefit from. There are some important and useful links there as well.

A lot of folks swipe the html source (right click and save as) on that page - use it as a template for liquid tabular layout if you like. I ask that folks leave a link to Radio Free Dexterdyne somewhere if they do - but most that have nicked the source are in a spot where it would be difficult to explain to their pay masters why they had to swipe a source in the first place.

That's alright by me too I guess, anyone (including me) who ever starts to brag that they never poked around a third party html source for their own sites is lying.

So - please brothers - could I again ask that this unfortunate thread be put to bed? I will refrain from offering WWW advise here in future so as not to stir the pot again. Thank you.

Reply to
Dexter J

MotorsForum website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.