How to do a good job of knowledge collection? What you need to learn to do web scraping

First, the basic knowledge of the production of the collector.

1. Understand the knowledge of some simple regular expressions, and know some basic knowledge is helpful for everyone to set rules in the collection software, some basic knowledge of regular expressions is easy to master, and most of the collection uses the most basic content of regular expressions.

Such as: at least to know"."n” “s” "d" "3-6]"。

To understand the basic meaning of these simple regular formulas, I will not explain them in detail here, but you can find information on the Internet. Personally, I think it's easier to learn, but if you don't use it often, you'll forget it more quickly. It is often the case that the collection rules are written while checking the rules on the Internet.

2. Some program commands will be used to crawl the content of the web page correctly.

Second, the idea of collecting the first station

Just take a look at how to collect the century, just know the ID number of a book and you can get started. Through the previous collection, scrape the data from the web page layer by layer, and then take out the required content according to the regular expression tags you set, determine whether it is text content or **content, and then write it to the back-end database respectively.

If it is a batch collection, it will be cycled according to the set ID number until all the collection is completed. If you set the collection to start from 1 50000, you can basically collect all the books at the target book station.

Even if there is an empty number in the target station, for example: 4678 has no book, so there will be an error when grabbing this number, and it will be done by judging that it is an empty number through regular collection.

The blue heart collection in hand has achieved three batch collection modes.

The basic idea of anti-collection.

At present, there is no one that does not have a completely thorough solution for anti-collection.

1. Increase the difficulty of the rules of collection, and often change the rules, this method is relatively easy to use to prevent collection, and the effect is better and easier.

2. IIS and Apache can be set to limit the collection of **.

IIS can be achieved by overwriting some add-on modules, but it is difficult to implement the method directly in IIS through settings, and add-on modules are required to do this.

Apache can achieve the most simple anti-mining and anti-hotlinking, the setup is relatively simple, and it is very practical.

Summary. Centralized procurement work is an important part of the national procurement work, with a strong purpose of improving procurement efficiency and effectively saving financial funds, but due to the uneven quality of procurement and procurement center personnel at all levels, most of them are less professional, the bureaucratic atmosphere is serious, and even serious corruption occurs frequently, resulting in the actual procurement work, the purpose of centralized procurement is counterproductive and difficult to achieve.

At present, the following points can be used to effectively improve this situation:

1. Strengthen the professional requirements of the staff of the procurement centralized procurement department, such as the certificate must be held, the relevant post personnel must obtain the tenderer certificate within a time limit, and the newly hired personnel will take the tenderer certificate as the basic requirement.

2. Reasonable and effective selection of outstanding social institutions that carry out procurement business in the local area, and advocate and recommend the cooperation between the organs and units at all levels and the entrusted bidding business of the selected outstanding social institutions, which is also in line with the spirit of the state to strengthen the purchase of social third-party services.

3. Optimize the bidding method and bidding process, and reduce the interference of human factors in centralized procurement, such as the purchase of large quantities of general goods, which can also use online price comparison and other methods that can effectively save money.

There are still many ideas that need to be formulated in combination with the actual situation of local centralized procurement, welcome to exchange.

Centralized procurement work is an important part of the national procurement work, with a strong purpose of improving procurement efficiency and effectively saving financial funds, but due to the uneven quality of procurement and procurement center personnel at all levels, most of them are less professional, the bureaucratic atmosphere is serious, and even serious corruption occurs frequently, resulting in the actual procurement work, the purpose of centralized procurement is counterproductive and difficult to achieve. At present, the methods that can effectively improve this situation can immediately think of the following points for your reference: 1. Strengthen the professional requirements of the staff of the procurement department, such as the professional requirements of the staff of the procurement department, such as the certificate must be held, and the relevant personnel must take the tenderer certificate within a time limit, and the newly hired personnel will take the tenderer certificate as the basic requirement.

2. Reasonable and effective selection of outstanding social institutions that carry out procurement business in the local area, and advocate and recommend the cooperation between the organs and units at all levels and the entrusted bidding business of the selected outstanding social institutions, which is also in line with the spirit of the state to strengthen the purchase of social third-party services. 3. Optimize the bidding method and bidding process, and reduce the interference of human factors in centralized procurement, such as the purchase of large quantities of general goods, which can also use online price comparison and other methods that can effectively save money. There are still many ideas that need to be formulated in combination with the actual situation of local centralized procurement, welcome to exchange.

PC-side collection tool

.Powerful text extension tool popclip

PopClip is a well-known gadget on Mac, and when it comes to PopClip, it's probably the best app to buy for Mac. Its operation is also very simple, you only need to select the text and then reverse the white text, this software is simple and efficient, with powerful expansion functions, when there is no plug-in installed, it has the following functions.

Paste the open link.

Duplicate the dictionary. Spell check.

Email redirects. Isn't that great? What's even more amazing is that it also supports more than a hundred different kinds of plugins that have many different features. For example, it supports ......Chinese word translation, modifying text formatting, searching Douban, saving to, and so on

With just one click of the Evernote button, you can create a new file in Evernote.

Powerful copy-paste tool paste

Paste combines so well with Mac that it looks like it's a native app. Not only can it record the history of copying, but the categories and previews are also very good, which is really a must-have artifact for writers!

Cheat shortcut query tool

Speaking of an efficient writer, most of them are keyboard parties, for example, I haven't used a mouse for nearly two years, because the usual operation can be completely solved with shortcuts, but some new software itself doesn't know its shortcuts. Groping one by one?

Not at all.

It's good to have cheet command.,Install it.,When you're using a software,Long press the command key to see the complete shortcut key mapping diagram Oh (can't take a screenshot.,You can only shoot it by hand),Use the software against the shortcut keys,Isn't it cool and crooked?

., the in-web shortcuts

If you want to exit, just press the esc key.

With this artifact, combined with the shortcut keys on the browser, you won't need the touch version at all when browsing the web! Productivity has been significantly improved!

2.Mobile capture tool

In addition to collecting data on the PC, we often need to collect some information and make some notes on the mobile side, in addition to pocket, there are some commonly used methods and software.

My Evernote

Love MoAt last.