Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

6 votes290 views3 answers
No matching distribution found for tensorflow==2.2.0
Python
0 vote272 views1 answer
How to create simple window in wxPython
Python
0 vote310 views1 answer
Python imread multiple images
Python
1 vote299 views1 answer
Reverse one hot encoding python numpy
Python
1 vote285 views1 answer
Module 'tensorflow_core.compat.v1.random' has no attribute 'set_seed'
Python
5 votes240 views2 answers
Jupyter convert cell to text
Python
0 vote271 views1 answer
Python check if 3 values are equal
Python
6 votes302 views2 answers
Pandas column not in list
Python
0 vote318 views1 answer
Wie printe ich in python
Python
1 vote313 views1 answer
Index of max in tensor
Python