The rules for formal and informal votes in the House of Representatives are specified in Section 268 of the Commonwealth Electoral Act.
Section 268 is shown below. It is also available on AUSTLII.
The rules for formal and informal votes in the Senate are outlined in Section 239 of the Commonwealth Electoral Act.
The Australian Electoral Commission has also issued a booklet: Ballot Paper Formality Guidelines
Commonwealth Electoral Act, Section 268
COMMONWEALTH ELECTORAL ACT 1918 – SECT 268
Informal ballot papers
(1) A ballot paper shall (except as otherwise provided by section 239, and by the regulations relating to voting by post) be informal if:
(a) subject to subsection (2), it is not authenticated by the initials of the presiding officer or a voting officer (within the meaning of Division 3 of Part XVA), or by the presence of the official mark;
(b) subject to section 269 and subsection 270(1), in a Senate election, it has no vote indicated on it, or it does not indicate the voter’s first preference for 1 candidate and the order of his or her preference for all the remaining candidates;
(c) in a House of Representatives election, it has no vote indicated on it, or it does not indicate the voter’s first preference for 1 candidate and an order of preference for all the remaining candidates:
Provided that, where the voter has indicated a first preference for 1 candidate and an order of preference for all the remaining candidates except 1 and the square opposite the name of that candidate has been left blank, it shall be deemed that the voter’s preference for that candidate is the voter’s last and that accordingly the voter has indicated an order of preference for all the candidates:
Provided further that, where there are 2 candidates only and the voter has indicated his or her vote by placing the figure 1 in the square opposite the name of 1 candidate and has left the other square blank or placed a figure other than 2 in it, the voter shall be deemed to have indicated an order of preference for all the candidates;
(d) it has upon it any mark or writing (not authorized by this Act or the regulations to be put upon it) by which, in the opinion of the Divisional Returning Officer, the voter can be identified:
Provided that paragraph (d) shall not apply to any mark or writing placed upon the ballot paper by an officer, notwithstanding that the placing of the mark or writing upon the ballot paper is a contravention of this Act; or
(e) in the case of an absent vote–the ballot paper is not contained in an envelope bearing a declaration made by the elector under subsection 222(1) or (1A).
(2) A ballot paper to which paragraph (1)(a) applies shall not be informal by virtue of that paragraph if the Divisional Returning Officer responsible for considering the question of the formality of the ballot paper is satisfied that it is an authentic ballot paper on which a voter has marked a vote and the officer has endorsed the ballot paper with the words ‘I am satisfied that this ballot paper is an authentic ballot paper on which a voter has marked a vote.’.
(3) A ballot paper shall not be informal for any reason other than the reasons specified in this section, but shall be given effect to according to the voter’s intention so far as that intention is clear.