Python ile CSV dosyasını okuma ve yazma

Writing data into different types of CSV files

In the previous article, we learned how to read csv files in Python. In similar way, we can also write a new or edit existing csv files in Python.
In Python we use csv.writer() module to write data into csv files. This module is similar to the csv.reader() module.

Writing on Existing File

We have a people.csv file with following data.
SN, Name, City
1, John, Washington
2, Eric, Los Angeles
3, Brad, Texas
Now, we are going to modify people.csv file.

Example 1: Modifying existing rows of people.csv

import csv

row = ['2', ' Marie', ' California']

with open('people.csv', 'r') as readFile:
reader = csv.reader(readFile)
lines = list(reader)
lines[2] = row

with open('people.csv', 'w') as writeFile:
    writer = csv.writer(writeFile)
    writer.writerows(lines)

readFile.close()
writeFile.close()
When we open the people.csv file with text editor, then it will show:
SN, Name, City
1, John, Washington
2, Marie, California
3, Brad, Texas
In the above program, we modified the third row of people.csv and saved the result. At first, we read the people.csv file using csv.reader() function. Then, we used list() function to convert all the csv data in a list and store in lines. After that, we changed third row of csv file with row i.e lines[2] = row. Finally, we write the values of lines list to people.csv file.
Sometimes, we may need to add new rows in the existing csv file. So, we are going to append a new row to people.csv file used in Example 1.

Example 2: Appending new rows to people.csv file

import csv

row = ['4', ' Danny', ' New York']

with open('people1.csv', 'a') as csvFile:
    writer = csv.writer(csvFile)
    writer.writerow(row)

csvFile.close()
When we open people.csv file with text editor, then it will show :
SN, Name, City
1, John, Washington
2, Marie, California
3, Brad, Texas
4, Danny, New York
In the above program, we append a new row into people.csv. For this, we opened the csv file in 'a' append mode. Then, we write the value of row after the last line of the people.csv file.

Normal CSV File

We create a normal csv file using writer() method of csv module having default delimiter comma(,).

Example 3: Write a python list into person.csv file

import csv

csvData = [['Person', 'Age'], ['Peter', '22'], ['Jasmine', '21'], ['Sam', '24']]

with open('person.csv', 'w') as csvFile:
    writer = csv.writer(csvFile)
    writer.writerows(csvData)

csvFile.close()
When we open the person.csv file with text editor, then it will show :
Person,Age
Peter,22
Jasmine,21
Sam,24
In the above program, we use csv.writer() function to write data from a list csvData into a csv file person.csv.
Note: The writerow() method writes one row at a time. If you need to write all the data at once you can use writerows() method.

CSV Files with quotes

We can write the csv file with quotes, by registering new dialects using csv.register_dialect() class of csv module.

Example 4 : Write into person1.csv file

import csv

person = [['SN', 'Person', 'DOB'],
['1', 'John', '18/1/1997'],
['2', 'Marie','19/2/1998'],
['3', 'Simon','20/3/1999'],
['4', 'Erik', '21/4/2000'],
['5', 'Ana', '22/5/2001']]

csv.register_dialect('myDialect',
quoting=csv.QUOTE_ALL,
skipinitialspace=True)

with open('person1.csv', 'w') as f:
    writer = csv.writer(f, dialect='myDialect')
    for row in person:
        writer.writerow(row)

f.close()
When we open person1.csv file, we get following output :
"SN","Person","DOB"
"1","John","18/1/1997"
"2","Marie","19/2/1998"
"3","Simon","20/3/1999"
"4","Erik","21/4/2000"
"5","Ana","22/5/2001"
In above program, we register a dialect named myDialect. Inside myDialect we use quoting=csv.QUOTE_ALL to write the double quote on all the values.

CSV files with Custom Delimiters


A delimiter is a string used to separate fields. The default value is comma(,).
We can write csv file having custom delimiter by registering a new dialect with the help of csv.register_dialect().

Example 5 : Writing with custom delimiter as pipe(|)

import csv

person = [['SN', 'Person', 'DOB'],
['1', 'John', '18/1/1997'],
['2', 'Marie','19/2/1998'],
['3', 'Simon','20/3/1999'],
['4', 'Erik', '21/4/2000'],
['5', 'Ana', '22/5/2001']]

csv.register_dialect('myDialect',
delimiter = '|',
quoting=csv.QUOTE_NONE,
skipinitialspace=True)

with open('dob.csv', 'w') as f:
    writer = csv.writer(f, dialect='myDialect')
    for row in person:
        writer.writerow(row)

f.close()
When we open dob.csv file, we get following output:
SN|Person|DOB
1|John|18/1/1997
2|Marie|19/2/1998
3|Simon|20/3/1999
4|Erik|21/4/2000
5|Ana|22/5/2001
In the above program, we register a dialect with delimiter as pipe(|). Then we write a list into a csv file dob.csv.

CSV File with a Lineterminator

A lineterminator is a string used to terminate lines produced by writer. The default value is \r\n.
We can write csv file with a lineterminator in Python by registering new dialects using csv.register_dialect() class of csv module

Example 6 : Writing csv file using a lineterminator

import csv

csvData = [['Fruit', 'Quantity'], ['Apple', '5'], ['Orange', '7'], ['Mango', '8']]

csv.register_dialect('myDialect', delimiter = '|', lineterminator = '\r\n\r\n')

with open('lineterminator.csv', 'w') as f:
    writer = csv.writer(f, dialect='myDialect')
    writer.writerows(csvData)

f.close()
When we open the lineterminator.csv file, we get following output:
Fruit|Quantity
Apple|5
Orange|7
Mango|8
In above code, we register a new dialects as myDialect. Then, we use delimiter='|' where a | is considered as column separator. After that, we use lineterminator='\r\n\r\n' where each row separates after every two lines.
Note : Python's CSV module only accepts \r\n\n or \r as lineterminator.

CSV File with quotechars

We can write the csv file with custom quote characters, by registering new dialects using csv.register_dialect() class of csv module.

Example 7: Writing CSV FIle with quotechars

import csv

csvData = [['SN', 'Items'], ['1', 'Pen'], ['2', 'Book'], ['3', 'Copy']]

csv.register_dialect('myDialect',
delimiter = '|',
quotechar = '"',
quoting=csv.QUOTE_ALL,
skipinitialspace=True)

with open('quotechars.csv', 'w') as csvFile:
    writer = csv.writer(csvFile, dialect='myDialect')
    writer.writerows(csvData)

print("writing completed")

csvFile.close()
Output:
"SN"|"Items"
"1"|"Pen"
"2"|"Book"
"3"|"Copy"
In the above program, we register a dialect called myDialect. Then we use delimiter as pipe(|) and quotechar as doublequote '"'.

Writing CSV file into a Dictionary

Using DictWriter() class of csv module, we can write a csv file into a dictionary. It works similar to the writer() function but creates an object which maps data into a dictionary. The keys are given by the fieldnames parameter.

Example 8: Writing dictionary into peak.csv file

import csv

data = [{'mountain' : 'Everest', 'height': '8848'},
      {'mountain' : 'K2 ', 'height': '8611'},
      {'mountain' : 'Kanchenjunga', 'height': '8586'}]

with open('peak.csv', 'w') as csvFile:
    fields = ['mountain', 'height']
    writer = csv.DictWriter(csvFile, fieldnames=fields)
    writer.writeheader()
    writer.writerows(data)

print("writing completed")

csvFile.close()
When we open peak.csv file, it will contain following output :
mountain,height
Everest,8848
K2,8611
Kangchenjunga,8586
In the above program, we use fieldnames as headings of each column in csv file. Then, we use a DictWriter() to write dictionary data into peak.csv file.

Example 9: Writing dictionary into grade.csv file with custom dialects

import csv

csv.register_dialect('myDialect', delimiter = '|', quoting=csv.QUOTE_ALL)

with open('grade.csv', 'w') as csvfile:
    fieldnames = ['Name', 'Grade']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames, dialect="myDialect")
    writer.writeheader()
    writer.writerows([{'Grade': 'B', 'Name': 'Alex'},
                 {'Grade': 'A', 'Name': 'Bin'},
                 {'Grade': 'C', 'Name': 'Tom'}])

print("writing completed")
When we open grade.csv file, it will contain following output:
"Name"|"Grade"
"Alex"|"B"
"Bin"|"A"
"Tom"|"C"
In the above program, we create a custom dialect called myDialect with pipe(|) as delimiter. Then, use fieldnames as headings of each column in csv file. Finally, we use a DictWriter()to write dictionary data into grade.csv file.


Kaynak : https://www.programiz.com/python-programming/working-csv-files#quotechars-files

İzlenmesi gereken 12 kritik Linux log dosyası



What are Linux log files

Log files are a set of records that Linux maintains for the administrators to keep track of important events. They contain messages about the server, including the kernel, services and applications running on it. 
Linux provides a centralized repository of log files that can be located under the  /var/log directory.
The log files generated in a Linux environment can typically be classified into four different categories:
  • Application Logs
  • Event Logs
  • Service Logs
  • System Logs

Why monitor Linux log files

Log management is an integral part of any server administrator’s responsibility.
By monitoring Linux log files, you can gain detailed insight on server performance, security, error messages and underlying issues by. If you want to take a proactive vs. a reactive approach to server management, regular log file analysis is 100% required.
In short, log files allow you to anticipate upcoming issues before they actually occur. 

Which Linux log files to monitor

Monitoring and analyzing all of them can be a challenging task.
The sheer volume of logs can sometimes make it frustrating just to drill down and find the right file that contains the desired information.
To make it a little easier for you, we will introduce you to some of the most critical Linux log files that you must be monitoring.
Note: Please note that this is not an all-inclusive list - but just a subset of the important log files that matter the most. The more you can handle, the better it is for the health of your server. Listed below are the bare minimum that you must monitor without fail

/var/log/messages

What’s logged here?:

  • This log file contains generic system activity logs.
  • It is mainly used to store informational and non-critical system messages.
  • In Debian-based systems,  /var/log/syslog directory serves the same purpose.

How can I use these logs?:

  • Here you can track non-kernel boot errors, application-related service errors and the messages that are logged during system startup.
  • This is the first log file that the Linux administrators should check if something goes wrong.
  • For example, you are facing some issues with the sound card. To check if something went wrong during the system startup process, you can have a look at the messages stored in this log file. 

/var/log/auth.log

What’s logged here?

  • All authentication related events in Debian and Ubuntu server are logged here.
  • If you’re looking for anything involving the user authorization mechanism, you can find it in this log file.

How can I use these logs?:

Suspect that there might have been a security breach in your server? Notice a suspicious javascript file where it shouldn’t be? If so, then find this log file asap!
  • Investigate failed login attempts
  • Investigate brute-force attacks and other vulnerabilities related to user authorization mechanism.

/var/log/secure

What’s logged here?

RedHat and CentOS based systems use this log file instead of /var/log/auth.log. 
  • It is mainly used to track the usage of authorization systems.
  • It stores all security related messages including authentication failures.
  • It also tracks sudo logins, SSH logins and other errors logged by system security services daemon.

How can I use these logs?:

  • All user authentication events are logged here.
  • This log file can provide detailed insight about unauthorized or failed login attempts
  • Can be very useful to detect possible hacking attempts.
  • It also stores information about successful logins and tracks the activities of valid users.

/var/log/boot.log

What’s logged here?

  • The system initialization script, /etc/init.d/bootmisc.sh, sends all bootup messages to this log file
  • This is the repository of booting related information and messages logged during system startup process.

How can I use these logs?:

  • You should analyze this log file to investigate issues related to improper shutdown, unplanned reboots or booting failures.
  • Can also be useful to determine the duration of system downtime caused by an unexpected shutdown.

/var/log/dmesg

What’s logged here?

  • This log file contains Kernel ring buffer messages.
  • Information related to hardware devices and their drivers are logged here.
  • As the kernel detects physical hardware devices associated with the server during the booting process, it captures the device status, hardware errors and other generic messages.

How can I use these logs?:

  • This log file is useful for dedicated server customers mostly.
  • If a certain hardware is functioning improperly or not getting detected, then you can rely on this log file to troubleshoot the issue.
  • Or, you can purchase a managed server from us and we’ll monitor it for you.

/var/log/kern.log

What’s logged here?

This is a very important log file as it contains information logged by the kernel.

How can I use these logs?:

  • Perfect for troubleshooting kernel related errors and warnings.
  • Kernel logs can be helpful to troubleshoot a custom-built kernel.
  • Can also come handy in debugging hardware and connectivity issues.

/var/log/faillog

What’s logged here?

This file contains information on failed login attempts.

How can I use these logs?:

It can be a useful log file to find out any attempted security breaches involving username/password hacking and brute-force attacks.

/var/log/cron

What’s logged here?

This log file records information on cron jobs.

How can I use these logs

  • Whenever a cron job runs, this log file records all relevant information including successful execution and error messages in case of failures.
  • If you’re having problems with your scheduled cron, you need to check out this log file.

/var/log/yum.log

What’s logged here?

It contains the information that is logged when a new package is installed using the yum command.

How can I use these logs?:

  • Track the installation of system components and software packages.
  • Check the messages logged here to see whether a package was correctly installed or not.
  • Helps you troubleshoot issues related to software installations.
Suppose your server is behaving unusually and you suspect a recently installed software package to be the root cause for this issue. In such cases, you can check this log file to find out the packages that were installed recently and identify the malfunctioning program. 

/var/log/
maillog or /var/log/mail.log

What’s logged here?

All mail server related logs are stored here.

How can I use these logs?

  • Find information about postfix, smtpd, MailScanner, SpamAssassain or any other email related services running on the mail server.
  • Track all the emails that were sent or received during a particular period
  • Investigate failed mail delivery issues.
  • Get information about possible spamming attempts blocked by the mail server.
  • Trace the origin of an incoming email by scrutinizing this log file.

var/log/httpd/

What’s logged here?

  • This directory contains the logs recorded by the Apache server.
  • Apache server logging information are stored in two different log files – error_log and access_log.

How can I use these logs?:

  • The error_log contains messages related to httpd errors such as memory issues and other system related errors.
  • This is the place where Apache server writes events and error records encountered while processing httpd requests.
  • If something goes wrong with the Apache webserver, check this log for diagnostic information.
  • Besides the error-log file, Apache also maintains a separate list of access_log.
  • All access requests received over HTTP are stored in the access_log file.
  • Helps you keep track of every page served and every file loaded by Apache.
  • Logs the IP address and user ID of all clients that make connection requests to the server.
  • Stores information about the status of the access requests, – whether a response was sent successfully or the request resulted in a failure.

/var/log/mysqld.log or /var/log/mysql.log

What’s logged here?

  • As the name suggests, this is the MySQL log file.
  • All debug, failure and success messages related to the [mysqld] and [mysqld_safe] daemon are logged to this file.
  • RedHat, CentOS and Fedora stores MySQL logs under  /var/log/mysqld.log, while Debian and Ubuntu maintains the log in /var/log/mysql.log directory.

How can I use this log?

  • Use this log to identify problems while starting, running, or stopping mysqld.
  • Get information about client connections to the MySQL data directory
  • You can also setup ‘long_query_time’ parameter to log information about query locks and slow running queries.

Final Takeaway

While monitoring and analyzing all the log files generated by the system can be a difficult task, you can make use of a centralized log monitoring tool to simplify the process.
Some of our customers take advantage of using  Nagios Log Server to manage their server logs. There are many opensource options available if that’s out of the budget. Needless to say though, monitoring Linux logs manually is hard.
So if you want to take a truly proactive approach to server management, investing in a centralized log collection and analysis platform which allows you to view log data in real-time and set up alerts to notify you when potential threats arise.

Kaynak: https://www.eurovps.com/blog/important-linux-log-files-you-must-be-monitoring/