From: Mike Kravetz
mainline inclusion
from mainline-v5.15-rc1
commit e32d20c0c88b1cd0a44f882c4f0eb2f536363d1b
category: bugfix
issue: #I4NRS5
CVE: NA
Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
Signed-off-by: Yu Changchun
----------------------------------------------------------------------
When removing a hugetlb page from the pool the ref count is set to one (as
the free page has no ref count) and compound page destructor is set to
NULL_COMPOUND_DTOR. Since a subsequent call to free the hugetlb page will
call __free_pages for non-gigantic pages and free_gigantic_page for
gigantic pages the destructor is not used.
However, consider the following race with code taking a speculative
reference on the page:
Thread 0 Thread 1
-------- --------
remove_hugetlb_page
set_page_refcounted(page);
set_compound_page_dtor(page,
NULL_COMPOUND_DTOR);
get_page_unless_zero(page)
__update_and_free_page
__free_pages(page,
huge_page_order(h));
/* Note that __free_pages() will simply drop
the reference to the page. */
put_page(page)
__put_compound_page()
destroy_compound_page
NULL_COMPOUND_DTOR
BUG: kernel NULL pointer
dereference, address:
0000000000000000
To address this race, set the dtor to the normal compound page dtor for
non-gigantic pages. The dtor for gigantic pages does not matter as
gigantic pages are changed from a compound page to 'just a group of pages'
before freeing. Hence, the destructor is not used.
Link: https://lkml.kernel.org/r/20210809184832.18342-4-mike.kravetz@oracle.com
Signed-off-by: Mike Kravetz
Reviewed-by: Muchun Song
Cc: Michal Hocko
Cc: Oscar Salvador
Cc: David Hildenbrand
Cc: Matthew Wilcox
Cc: Naoya Horiguchi
Cc: Mina Almasry
Signed-off-by: Andrew Morton
Signed-off-by: Linus Torvalds
Signed-off-by: Chen Wandun
Reviewed-by: Kefeng Wang
Signed-off-by: Zheng Zengkai
Conflicts:
mm/hugetlb.c
[use update_and_free_page instead of remove_hugetlb_page]
Signed-off-by: Yu Changchun
---
mm/hugetlb.c | 20 +++++++++++++++++++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 17679e80fbc4..4ecf9a0622df 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1353,9 +1353,26 @@ static void update_and_free_page(struct hstate *h, struct page *page)
}
VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page);
VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page);
- set_compound_page_dtor(page, NULL_COMPOUND_DTOR);
+ /*
+ * Very subtle
+ *
+ * For non-gigantic pages set the destructor to the normal compound
+ * page dtor. This is needed in case someone takes an additional
+ * temporary ref to the page, and freeing is delayed until they drop
+ * their reference.
+ *
+ * For gigantic pages set the destructor to the null dtor. This
+ * destructor will never be called. Before freeing the gigantic
+ * page destroy_compound_gigantic_page will turn the compound page
+ * into a simple group of pages. After this the destructor does not
+ * apply.
+ *
+ * This handles the case where more than one ref is held when and
+ * after update_and_free_page is called.
+ */
set_page_refcounted(page);
if (hstate_is_gigantic(h)) {
+ set_compound_page_dtor(page, NULL_COMPOUND_DTOR);
/*
* Temporarily drop the hugetlb_lock, because
* we might block in free_gigantic_page().
@@ -1365,6 +1382,7 @@ static void update_and_free_page(struct hstate *h, struct page *page)
free_gigantic_page(page, huge_page_order(h));
spin_lock(&hugetlb_lock);
} else {
+ set_compound_page_dtor(page, COMPOUND_PAGE_DTOR);
__free_pages(page, huge_page_order(h));
}
}
--
2.25.1
