Large CSV Manipulation tricks

I sometimes have to deal with large csv files that need some massaging and they are too big to rely on spreadsheet program to handle.  Here are some tips that I sometimes use in the terminal on ubuntu.

To remove duplicate rows:

awk '!seen[$0]++' duped.csv > DEduped.csv

To remove rows with certain strings in the row and make a backup file:

Steps to set up a private freenode channel

THe followng steps will quickly setup a freenode IRC channel.

This requires your nick is properly registered

/msg ChanServ info #myChannelName

If it does not exist, proceed.

/join #myChannelName
/msg ChanServ register #myChannelName

Password protect it

/msg ChanServ set #myChannelName mlock +k thePassword

If / when it comes time to delete the channel:

/msg ChanServ DROP #myChannelName

Computer Workspace Accessories

I'm a full-time web developer so I spend a LOT of time in front of my computer. In addition to my convertible stand-up sit-down desk I have several things that make that time healthier and more comfortable.

Subscribe to RSS - other