Asked  7 Months ago    Answers:  5   Viewed   32 times

My server is having unusually high CPU usage, and I can see Apache is using way too much memory. I have a feeling, I'm being DOS'd by a single IP - maybe you can help me find him?

I've used the following line, to find the 10 most "active" IPs:

cat access.log | awk '{print $1}' |sort  |uniq -c |sort -n |tail

The top 5 IPs have about 200 times as many requests to the server, as the "average" user. However, I can't find out if these 5 are just very frequent visitors, or they are attacking the servers.

Is there are way, to specify the above search to a time interval, eg. the last two hours OR between 10-12 today?

Cheers!

UPDATED 23 OCT 2011 - The commands I needed:

Get entries within last X hours [Here two hours]

awk -vDate=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` ' { if ($4 > Date) print Date FS $4}' access.log

Get most active IPs within the last X hours [Here two hours]

awk -vDate=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` ' { if ($4 > Date) print $1}' access.log | sort  |uniq -c |sort -n | tail

Get entries within relative timespan

awk -vDate=`date -d'now-4 hours' +[%d/%b/%Y:%H:%M:%S` -vDate2=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` ' { if ($4 > Date && $4 < Date2) print Date FS Date2 FS $4}' access.log

Get entries within absolute timespan

awk -vDate=`date -d '13:20' +[%d/%b/%Y:%H:%M:%S` -vDate2=`date -d'13:30' +[%d/%b/%Y:%H:%M:%S` ' { if ($4 > Date && $4 < Date2) print $0}' access.log 

Get most active IPs within absolute timespan

awk -vDate=`date -d '13:20' +[%d/%b/%Y:%H:%M:%S` -vDate2=`date -d'13:30' +[%d/%b/%Y:%H:%M:%S` ' { if ($4 > Date && $4 < Date2) print $1}' access.log | sort  |uniq -c |sort -n | tail

 Answers

31

yes, there are multiple ways to do this. Here is how I would go about this. For starters, no need to pipe the output of cat, just open the log file with awk.

awk -vDate=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print Date, $0}' access_log

assuming your log looks like mine (they're configurable) than the date is stored in field 4. and is bracketed. What I am doing above is finding everything within the last 2 hours. Note the -d'now-2 hours' or translated literally now minus 2 hours which for me looks something like this: [10/Oct/2011:08:55:23

So what I am doing is storing the formatted value of two hours ago and comparing against field four. The conditional expression should be straight forward.I am then printing the Date, followed by the Output Field Separator (OFS -- or space in this case) followed by the whole line $0. You could use your previous expression and just print $1 (the ip addresses)

awk -vDate=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` '$4 > Date {print $1}' | sort  |uniq -c |sort -n | tail

If you wanted to use a range specify two date variables and construct your expression appropriately.

so if you wanted do find something between 2-4hrs ago your expression might looks something like this

awk -vDate=`date -d'now-4 hours' +[%d/%b/%Y:%H:%M:%S` -vDate2=`date -d'now-2 hours' +[%d/%b/%Y:%H:%M:%S` '$4 > Date && $4 < Date2 {print Date, Date2, $4} access_log'

Here is a question I answered regarding dates in bash you might find helpful. Print date for the monday of the current week (in bash)

Tuesday, June 1, 2021
 
Sendy
answered 7 Months ago
99

Apache 2.4.3 (or maybe slightly earlier) added a new security feature that often results in this error. You would also see a log message of the form "client denied by server configuration". The feature is requiring a user identity to access a directory. It is turned on by DEFAULT in the httpd.conf that ships with Apache. You can see the enabling of the feature with the directive

Require all denied

This basically says to deny access to all users. To fix this problem, either remove the denied directive (or much better) add the following directive to the directories you want to grant access to:

Require all granted

as in

<Directory "your directory here">
   Order allow,deny
   Allow from all
   # New directive needed in Apache 2.4.3: 
   Require all granted
</Directory>
Wednesday, June 9, 2021
 
avon_verma
answered 6 Months ago
53

The following solutions are applicable since spark 1.5 :

For lower than :

// filter data where the date is lesser than 2015-03-14
data.filter(data("date").lt(lit("2015-03-14")))      

For greater than :

// filter data where the date is greater than 2015-03-14
data.filter(data("date").gt(lit("2015-03-14"))) 

For equality, you can use either equalTo or === :

data.filter(data("date") === lit("2015-03-14"))

If your DataFrame date column is of type StringType, you can convert it using the to_date function :

// filter data where the date is greater than 2015-03-14
data.filter(to_date(data("date")).gt(lit("2015-03-14"))) 

You can also filter according to a year using the year function :

// filter data where year is greater or equal to 2016
data.filter(year($"date").geq(lit(2016))) 
Thursday, June 10, 2021
 
Hat
answered 6 Months ago
Hat
55

It was because of the line 'WSGIPythonHome /usr/bin/pytyon3.4' in /etc/apache2/httpd.conf.

Without this line, it runs without error thank you

Sunday, August 22, 2021
 
mschuett
answered 4 Months ago
70

Since you have not mentioned about any errors, if START_DATE and END_DATE are DATETIME data type, there is nothing wrong with your query. If you are not getting the correct records, Please check the data.

However your date format may trouble you in a different server. There are some good practices you could adhere to avoid such issues.

-Whenever date is used as a string, try to use it in ISO or ISO8601 format (ie 'yyyymmdd' or 'yyyy-mm-ddThh:mi:ss.mmm')

-Also avoid joining tables with WHERE Table1, Table2 which is old and obsolete. JOINs are much better performed, neat and tidy.

You can change your query as follows;

SELECT DISTINCT T1.column1, T1.column2, T2.START_DATE, T2.END_DATE
FROM Table1 T1 JOIN Table2 T2 ON T1.column1 = T2.column2
WHERE (T2.START_DATE >= '20130115 10:58:58' AND 
       T2.END_DATE <= '20130118 10:58:58') 
ORDER BY T2.START_DATE DESC
Monday, November 8, 2021
 
rkeet
answered 3 Weeks ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :
 
Share