python - How to combine data from two different pages in scrapy and store them together

Question

1 For example, if there is a blog list page a, get the article list
2 Blog content details page b, get the article text
3 How to get the article content to the database in sequence?
4 The current problem is mainly that I don’t know How to combine data from 2 different pages

Answer 1

You can use request.meta to store additional information, such as

def parse_x1(self, response):
    extra = {}
    ...
    req = scrapy.Request(url2)
    req.callback = self.parse_x2
    ...
    req.meta['extra'] = extra 
    yield req
 
def parse_x2(self, response):
    item = {}
    extra = response.meta['extra']
    item.update(extra)
    ...
    
    yield item